On Tue, Aug 4, 2020 at 8:17 AM Ricky Teachey wrote:

The following comment is from the thread about adding kwd arg support to the square bracket operator (eg, `Vec = Dict[i=float, j=float]`).

On Tue, Aug 4, 2020, 2:57 AM Greg Ewing wrote:

On 4/08/20 1:16 pm, Steven D'Aprano wrote:

...

Why would we want to even consider a new approach to handling keyword arguments which applies only to three dunder methods, `__getitem__`, `__setitem__` and `__delitem__`, instead of handling keyword arguments in the same way that every other method handles them?

These methods are already kind of screwy in that they don't handle *positional* arguments in the usual way -- packing them into a tuple instead of passing them as individual arguments.

I think this is messing up everyone's intuition on how indexing should be extended to incorporate keyword args, or even whether this should be done at all.

-- Greg

So here is the main question of this thread:

Is there really not a backwards compatible, creative way a transition to positional args from a tuple in the item dunders could not be accomplished?

It's not my intention to champion any specific idea here, just creating a specific space for ideas to be suggested and mulled over.

There could be several reasons for changing the item dunder signatures. The immediate reason is making the intuition around how to add kwd arg support to square brackets more obvious and sane.

A second reason is it might be more intuitive for users who have to learn and remember that multiple arguments to [ ] get packed into a tuple, but this doesn't happen anywhere else.

My main issue with this is that, in my opinion, dunders are not something a beginner should be messing with anyway. By the time someone is experienced enough to start working on this, they are also experienced enough to understand that special cases like this exist for historical reasons.

Another reason: it could make writing code for specialized libraries that tend to abuse (for the good of us all!) item dunders, like pandas, much easier. Right now such libraries have to rely on their own efforts to break up a key:

def __getitem__(self, key): try: k1, k2 = key except TypeError: raise TypeError("two tuple key required")

But for regular function calls (as opposed to item getting) we get to write our signature however we want and rely on the language to handle all of this for us:

def f(k1, k2): # no worries about parsing out the arguments

But this is still a pretty simple piece of code. Is it worth having everyone start over from scratch to avoid dealing with 4 lines of code? Especially since knowing the number of indices ahead of time is a special case for a small number of projects like pandas. In most cases, the number of indices cannot be known until runtime, so this would provide no practical benefit for most projects.

-----------

One idea: change the "real" names of the dunders. Give `type` default versions of the new dunders that direct the call to the old dunder names.

The new get and del dunders would have behavior and signatures like (I am including **__kwargs since that could be an option in the future) :

def __getx__(self, /, *__key, **__kwargs): return self.__getitem__(__key, **__kwargs)

def __delx__(self, /,, *__key, **__kwargs): del self.__delitem__(__key, **__kwargs)

However the set dunder signature would be a problem, because to mirror the current behavior we end up writing what is now a syntax error:

def __setx__(self, /, *__key, __value, **__kwargs): self.__setitem__(__key, __value, **__kwargs)

The intended meaning above would be that the last positional argument gets assigned to __value. Maybe someone could suggest a way to fix this.

The simplest way would be to put "value" first: def __setx__(self, __value, /, *__key, **__kwargs):

On 2020-08-04 at 10:58:51 -0400, Todd wrote:

My main issue with this is that, in my opinion, dunders are not something a beginner should be messing with anyway. By the time someone is experienced enough to start working on this, they are also experienced enough to understand that special cases like this exist for historical reasons.

Ouch. Who am I to tell beginners what they should and shouldn't be messing with, and in what order? IMNSHO, history (let alone understanding it) comes *way* after writing a few dunders, even if you don't count __init__ as a dunder. Special cases aren't special enough to break the rules.

On Wed, Aug 5, 2020, 10:38 <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:

On 2020-08-04 at 10:58:51 -0400, Todd wrote:

...

My main issue with this is that, in my opinion, dunders are not something a beginner should be messing with anyway. By the time someone is experienced enough to start working on this, they are also experienced enough to understand that special cases like this exist for historical reasons.

Ouch.

Who am I to tell beginners what they should and shouldn't be messing with, and in what order? IMNSHO, history (let alone understanding it) comes *way* after writing a few dunders, even if you don't count __init__ as a dunder.

I should have been more clear, I was talking about these specific dunder methods. Overriding indexing isn't something that people will typically deal with before understanding the sequences indexing acts on, which is all they really need to understand to make sense of the current API. And history is hard to avoid. Why is it "def" instead of "function"? Why do dunder methods use "__" at all? Why does indexing use 0 instead of 1? Some things they are just going to need to accept are the way they are, and these are all things people are almost certainly going to need to encounter before making their own classes. Special cases aren't special enough to break the rules.

But they're aren't really any rules being broken here. It may be a sub-optimal solution in this case, but not a forbidden or even uncommon approach.

On Fri, Aug 07, 2020 at 05:54:18PM +1000, Steven D'Aprano wrote:

This proposal doesn't say anything about reversing the decision made all those years ago to bundle all positional arguments in a subscript into a single positional parameter. What's done is done, that's not going to change.

Sorry, I was referring to the proposal that inspired this thread, to add keyword arguments to subscripting. There's an actual concrete use-case for adding this, specifically for typing annotations, and I cannot help but feel that this thread is derailing the conversation to something that has not been requested by anyone actually affected by it. I may have allowed my frustration to run ahead of me, sorry. There is a tonne of code that relies on subscripting positional arguments to be bundled into a single parameter. Even if we agreed that this was suboptimal, and I don't because I don't know the rationale for doing it in the first place, I would be very surprised if the Steering Council gave the go-ahead to a major disruption and complication to the language just for the sake of making subscript dunders like other functions. Things would be different if, say, numpy or pandas or other heavy users of subscripting said "we want the short term churn and pain for long term benefit". But unless that happens, I feel this is just a case of piggy-backing a large, disruptive change of minimal benefit onto a a small, focused change, which tends to ruin the chances of the small change. So please excuse my frustration, I will try to be less grumpy about it. -- Steven

On 8/08/20 4:09 am, Ricky Teachey wrote:

If that incongruity were to be fixed, it seems to me it would become *obvious* that the semantic meaning of ` m[1, 2, a=3, b=2]` should definitely be:

m.__get__(1, 2, a=3, b=4)

It would certainly achieve that goal. The question is whether it would be worth the *enormous* upheaval of replacing the whole __getitem__ protocol. It's hard to overstate what a big deal that would be. The old protocol would still be with us, complicating everything unnecessarily, for a very long time. It took Python 3 to finally get rid of the __getslice__ protocol, and I don't think anyone has the appetite for a Python 4 any time soon. -- Greg

On Fri, Aug 7, 2020 at 9:12 AM Ricky Teachey wrote:

On Fri, Aug 7, 2020 at 4:19 AM Steven D'Aprano wrote:

...

On Fri, Aug 07, 2020 at 05:54:18PM +1000, Steven D'Aprano wrote:

...

This proposal doesn't say anything about reversing the decision made all those years ago to bundle all positional arguments in a subscript into a single positional parameter. What's done is done, that's not going to change.

Sorry, I was referring to the proposal that inspired this thread, to add keyword arguments to subscripting. There's an actual concrete use-case for adding this, specifically for typing annotations, and I cannot help but feel that this thread is derailing the conversation to something that has not been requested by anyone actually affected by it.

Well I wonder if they haven't asked because it would be such a huge change, and it seems unlikely to happen. But I surely don't know enough about the implementation details of these libraries to be able to say for certain one way or the other.

NumPy and pandas both care a lot about backwards compatibility, and don't like churn for the sake of churn. Speaking as someone who has been contributing to these libraries for years, the present syntax for positional arguments in __getitem__ is not a big deal, and certainly isn't worth breaking backwards compatibility over. In practice, this is worked around with a simple normalization step, e.g., something like: def __getitem__(self, key): if not isinstance(key, tuple): key = (key,) # normal __getitem__ method This precludes incompatible definitions for x[(1, 2)] and x[1, 2], but really, nobody this minor inconsistency is not a big deal. It is easy to write x[(1, 2), :] if you want to indicate a tuple for indexing along the first axis of an array. From my perspective, the other reasonable way to add keyword arguments to indexing would be in a completely backwards compatible with **kwargs.

I may have allowed my frustration to run ahead of me, sorry.

...

There is a tonne of code that relies on subscripting positional arguments to be bundled into a single parameter. Even if we agreed that this was suboptimal, and I don't because I don't know the rationale for doing it in the first place, I would be very surprised if the Steering Council gave the go-ahead to a major disruption and complication to the language just for the sake of making subscript dunders like other functions.

...

Things would be different if, say, numpy or pandas or other heavy users of subscripting said "we want the short term churn and pain for long term benefit".

But unless that happens, I feel this is just a case of piggy-backing a large, disruptive change of minimal benefit onto a a small, focused change, which tends to ruin the chances of the small change. So please excuse my frustration, I will try to be less grumpy about it.

I understand the grumpiness given your explanation. I'm really not wanting to derail that kwd args proposal-- I really like it, whatever the semantics of it turn out to be.

I was actually trying to help the kwd arg case here. As illustrated by the quote I included from Greg Ewing, there seems to be not even close to a consensus over what the semantic meaning of this should be:

m[1, 2, a=3, b=2]

Which could be made to mean one of the following things, or another thing I haven't considered:

1. m.__get__((1, 2), a=3, b=4) # handling of positional arguments unchanged from current behavior 2. m.__get__(1, 2, a=3, b=4) # change positional argument handling from current behavior 3. m.__get__((1, 2), {'a': 3, 'b': 4}) # handling of positional arguments unchanged from current behavior 4. m.__get__(KeyObject( (1, 2), {'a': 3, 'b': 4} )) # change positional argument handling from current behavior only in the case that kwd args are provided

As Greg said:

These methods are already kind of screwy in that they don't

...

handle *positional* arguments in the usual way -- packing them into a tuple instead of passing them as individual arguments. I think this is messing up everyone's intuition on how indexing should be extended to incorporate keyword args, or even whether this should be done at all.

To illustrate the comments of "kind of screwy" and "the usual way", using semantic meaning # 1 above, then these are totally equivalent * :

m[1, 2, a=3, b=4] m[(1, 2), a=3, b=4]

...even though these are totally different:

f(1, 2, a=3, b=4) f((1, 2), a=3, b=4)

So my intention here isn't to derail, but to help the kwd argument proposal along by solving this screwiness problem.

It is to suggest that maybe a way forward-- to make the intuition of the semantics of kwd args to [ ] much more obvious-- would be to change the signature so that this incongruity between what happens with "normal" method calls and the "call" for item-get/set/del can be smoothed out.

If that incongruity were to be fixed, it seems to me it would become *obvious* that the semantic meaning of ` m[1, 2, a=3, b=2]` should definitely be:

m.__get__(1, 2, a=3, b=4)

But if all of this is not helping but hindering. I am happy to withdraw the idea.

* Citing my source: I borrowed these examples from Jonathan Fine's message in the other thread

--- Ricky.

"I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BJOW5F... Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Aug 07, 2020 at 12:09:28PM -0400, Ricky Teachey wrote:

I was actually trying to help the kwd arg case here. As illustrated by the quote I included from Greg Ewing, there seems to be not even close to a consensus over what the semantic meaning of this should be:

m[1, 2, a=3, b=2]

This is Python-Ideas. If we asked for a consensus on what the print function should do, I'm sure we would find at least one person seriously insist that it ought to erase your hard drive *wink* But seriously, getting consensus is difficult, especially when people seem to be unwilling or unable to articulate why they prefer one behaviour over another, or the advantages vs disadvantages of a proposal.

Which could be made to mean one of the following things, or another thing I haven't considered:

1. m.__get__((1, 2), a=3, b=4) # handling of positional arguments unchanged from current behavior

By the way, I assume you meant `__getitem__` in each of your examples, since `__get__` is part of the descriptor protocol. Advantages: (1) Existing positional only subscripting does not change (backwards compatible). (2) Easy to handle keyword arguments. (3) Those who want to bundle all their keywords into a single object can just define a single `**kw` parameter. (4) Probably requires little special handling in the interpreter? (5) Probably requires the minimum amount of implementation effort? (6) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing. Disadvantages: none that I can see. (Assuming we agree that this is a useful feature.)

2. m.__get__(1, 2, a=3, b=4) # change positional argument handling from current behavior

Advantages: 1. Consistency with other methods and functions. Disadvantages: 1. Breaks backwards compatibility. 2. Will require a long and painful transition period during which time libraries will have to somehow support both calling conventions.

3. m.__get__((1, 2), {'a': 3, 'b': 4}) # handling of positional arguments unchanged from current behavior

I assume that if there are no keyword arguments given, only the first argument is passed to the method (as opposed to passing an empty dict). If not, the advantages listed below disappear. Advantages: (1) Existing positional only subscripting does not change (backwards compatible). (2) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing. Disadvantages: (1) Forces people to do their own parsing of keyword arguments to local variables inside the method, instead of allowing the interpreter to do it. (2) Compounds the "Special case breaks the rules" of subscript methods to keyword arguments as well as positional arguments. (3) It's not really clear to me that anyone actually wants this, apart from just suggesting it as an option. What's the concrete use-case for this?

4. m.__get__(KeyObject( (1, 2), {'a': 3, 'b': 4} )) # change positional argument handling from current behavior only in the case that kwd args are provided

Use-case: you want to wrap an arbitrary number of positional arguments, plus an arbitrary set of keyword arguments, into a single hashable "key object", for some unstated reason, and be able to store that key object into a dict. Advantage (double-edged, possible): (1) Requires no change to the method signature to support keyword parameters (whether you want them or not, you will get them). Disadvantages: (1) If you don't want keyword parameters in your subscript methods, you can't just *do nothing* and have them be a TypeError, you have to explicitly check for a KeyObject argument and raise: def __getitem__(self, index): if isinstance(item, KeyObject): raise TypeError('MyClass index takes no keyword arguments') (2) Seems to be a completely artificial and useless use-case to me. If there is a concrete use-case for this, either I have missed it, (in which case my apologies) or Jonathan seems to be unwilling or unable to give it. But if you really wanted it, you could get it with this signature and a single line in the body: def __getitem__(self, *args, **kw): key = KeyObject(*args, **kw) (3) Forces those who want named keyword parameters to parse them from the KeyObject value themselves. Since named keyword parameters are surely going to be the most common use-case (just as they are for other functions), this makes the common case difficult and the rare and unusual case easy. (4) KeyObject doesn't exist. We would need a new builtin type to support this, as well as the new syntax. This increases the complexity and maintenance burden of this new feature. (5) Compounds the "kind of screwy" (Greg's words) nature of subscripting by extending it to keyword arguments as well as positional arguments. -- Steven

On Fri, Aug 7, 2020 at 10:47 PM Steven D'Aprano wrote:

On Fri, Aug 07, 2020 at 12:09:28PM -0400, Ricky Teachey wrote:

...

I was actually trying to help the kwd arg case here. As illustrated by the quote I included from Greg Ewing, there seems to be not even close to a consensus over what the semantic meaning of this should be:

m[1, 2, a=3, b=2]

This is Python-Ideas. If we asked for a consensus on what the print function should do, I'm sure we would find at least one person seriously insist that it ought to erase your hard drive *wink*

Yeah I get it. Thanks. I just noticed it didn't seem like anything close to a consensus was even sort of rising to the surface.

By the way, I assume you meant `__getitem__` in each of your examples, since `__get__` is part of the descriptor protocol.

Whoops thanks for the correction. And thanks for being less grumpy. And thank you also for going through the 4 options I gave. I accept and agree with you on all of them. But I had forgotten the fifth. The semantic meaning of m[1, 2, a=3, b=2] might be made to mean: 5. m.__getx__(1, 2, a=3, b=4) ...which would in turn call, by default: m.__getitem__((1, 2), a=3, b=4) I was a little bit more detailed about in my first message so I'll quote that:

-----------

One idea: change the "real" names of the dunders. Give `type` default versions of the new dunders that direct the call to the old dunder names.

The new get and del dunders would have behavior and signatures like (I am including **__kwargs since that could be an option in the future) :

def __getx__(self, /, *__key, **__kwargs): return self.__getitem__(__key, **__kwargs)

def __delx__(self, /,, *__key, **__kwargs): del self.__delitem__(__key, **__kwargs)

However the set dunder signature would be a problem, because to mirror the current behavior we end up writing what is now a syntax error:

def __setx__(self, /, *__key, __value, **__kwargs): self.__setitem__(__key, __value, **__kwargs)

When overriding `__getx__` et al, you would always call super() on the __getx__ method, never use super().__getitem__: class My: def __getx__(self, my_arg, *args, my_kwarg, **kwargs): # the way I have written things this super call will cause a recursion error v = super().__getx__(*args, **kwargs) return combine(my_arg, my_kwarg, v) Many of the advantages are shared with #1 as you recounted them: (1) Existing positional only subscripting does not have to change for any existing code (backwards compatible). (2) Easy to handle keyword arguments. (3) Those who want to bundle all their keywords into a single object can just define a single `**kw` parameter. (4) Probably requires little special handling in the interpreter? (5) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing. (6). Consistency with other methods and functions for those that want to use the new dunders. Disadvantages: (1) Probably requires more implementation effort than #1 (2) Similar to #2 will also create a long transition period- but hopefully quite a bit less painful than just outright changing the signature of __getitem__ etc. However libraries do not have to support both calling conventions at all. They should be encouraged to start using the new one, but the old one will continue to work, perhaps perpetually. But maybe things would eventually get to the point that it could be eventually done away with. (3) Creates a new "kind of screwy" (Greg's words) situation that will at times need to be explained and understood. (4) Creates a sort of dual MRO for square bracket usage. You would end up with situations like this: class A: def __getitem__(self, key, **kwargs): print("A") class B(A): def __getitem__(self, key, **kwargs): print("B") super().__getitem__(key, **kwargs) class C(B): def __getx__(self, *key, **kwargs): print("C") super().__getx__(*key, **kwargs) class D(C): def __getx__(self, *key, **kwargs): print("D") super().__getx__(*key, **kwargs)

...

...

D()[None] D C B A

This code obviously looks just a little bit odd but the result is fine. However when different libraries and classes start getting mashed together over time, you might end up with a situation like this: class A: def __getitem__(self, key, **kwargs): print("A") class B(A): def __getx__(self, key, **kwargs): print("B") super().__getx__(*key, **kwargs) class C(B): def __getitem__(self, key, **kwargs): print("C") super().__getitem__(key, **kwargs) class D(C): def __getx__(self, *key, **kwargs): print("D") super().__getx__(*key, **kwargs)

...

...

D()[None] D B C A

Seems like this could be easily fixed with a class decorator, or perhaps the language could know to call the methods in the right order (which should be possible so long as the rule to never do a super().__getitem__ call inside of the new __getx__ method, and vice versa: don't super()__getx__ from a __getitem__ method_, is followed). Also Steven please ignore the messages I accidentally sent just to your email; apologies.

I misunderstood thanks for the clarification and if an apology is appropriate I gladly offer mine. On Sat, Aug 8, 2020, 9:01 PM Edwin Zimmerman wrote:

On 8/7/2020 10:47 PM, Ricky Teachey wrote:

On Fri, Aug 7, 2020 at 9:25 PM Edwin Zimmerman wrote:

...

On 8/7/2020 8:28 PM, Greg Ewing wrote:

...

I don't think anyone has the appetite for a Python 4 any time soon.

I'm included in "anyone" here. From reading this list, it seems to me that "Python 4" is invoked as some folks favorite magical justification for proposing major breaking changes. Python 3 works quite well, I think. Non-breaking, incremental changes suite me much better that large breaking ones. I have better things to do with my time than complete software rewrites of all the software projects I work on.

Nobody is asking you to rewrite anything in this thread.

Thankfully not. I did not intend to come across as commenting on this thread in particular, but rather as voicing a general comment against large breaking changes. Apparently, I wasn't clear enough about my intent.

--Edwin

On Sun, Aug 23, 2020 at 10:48 AM Stefano Borini wrote:

When you have a getitem operation, you are acting on a set of axes. e.g. a[4,5,6] acts on three axes. The first axis index is 4, the second is 5 and the third is 6. These axes currently are anonymous, but the whole idea is that a name could be assigned to them.

Which is kind of asymmetric with the whole args/kwargs structure of a function. In a function, your "axes" (which are your arguments) _always_ have a name. Not so in getitem operations: naming axes is optional. There's no such thing as optionally named function arguments.

Given the asymmetry, and the need for backward compat, would it make a possible solution to have __getitem__() accept one additional argument "names" containing a tuple with the names?

e.g. if you call a[1,2] __getitem__(index, names) will receive index=(1,2), names=(None, None) if you call a[foo=1, bar=2] __getitem__ will receive index=(1,2), names=("foo", "bar") if you call a[1, bar=2] __getitem__ will receive index=(1,2), names=(None, "bar")

Now, I personally don't like this solution, especially because now passing names depend if it was declared in the signature to begin with, but I am just throwing also this idea in the mix. Apologies if it was already passed by someone else.

My point is that the core of the issue is to name axes (with loose definition of what axes are. In the case of generic types, they are the degrees of freedom of the type). How these names are then handled (and recognised) _could_ be put in the hands of the user (in other words, python will say "here are the names, I have no idea if they mean something or not. it's your duty to find out, if you care about it")

It's a way of handling it I for one haven't seen suggested yet. I don't think I like it either. I suppose a trouble with the basis of this is the assumption that most users will be using what's inside a [ ] to define axes. Maybe this is correct. But it could easily be that the most popular usage of providing these named indices will be more akin to named arguments. Indeed, nearly all of the discussion (that I've seen at least) has seemed to presuppose that adding the ability to provide these named indices is really going to be more akin to a function call. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

Here is another way forward-- inspired by a conversation off-list with Jonathan Fine. I am calling it "signature dependent semantics". Right now, the semantic meaning of a __setitem__ function like this: # using ambiguous names for the parameters on purpose def __setitem__ (self, a, b): ... ...is currently as follows: *Note: below the === is not supposed to be code - I am using it as a way to state the semantics on the RHS of the signature on the LHS.* SIGNATURE === SEMANTICS (self, a, b) === (self, key_tuple, value) In the above, a on the left side, semantically, is the key tuple, and b in the value on the RHS. So right now, line 1 below calls line 2: [1]: d[1, 2] = foo [2]: d.__setitem__(key_tuple, value) And the call occurs this way: d.__setitem__((1,2), foo) So far, all of this is just a description of what currently happens. The signature dependent semantics proposal is NOT to change the semantic meaning of the above code in any way. These semantics would be maintained. Signature dependent semantics, as the name suggests, would change the semantic meaning of the __setitem__ signature in the case that more than two parameters are given in the signature. In other words, if a signature is provided like this, with 3 or more arguments: def: __setitem__(self, a, b, c): ... ...then in that case, the language would know there is a different semantic meaning intended. Right now, the above signature would raise a TypeError:

...

...

class D: ... def __setitem__(self, a, b, c): ... ... ... d=D() d[1,2] = object() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: __setitem__() missing 1 required positional argument: 'c'

However, using signature dependent semantics, the language would know to call using semantics like this instead of raising a TypeError: d.__setitem__(foo, 1, 2) # value is first positional argument In other words, in the case that there are three or more arguments present in the __getitem__ signature, the semantics of the signature CHANGES: SIGNATURE === CURRENT SEMANTICS: where key_tuple on the RHS contains (a,b), and value on the RHS is just c (self, a, b, c) === (self, key_tuple, value, CURRENTLY_UNUSED_PARAMETER) SIGNATURE === NEW SEMANTICS: a is value, b is positional argument 1, c is positional argument 2 (self, a, b, c) === (self, value, pos1, pos2) I'm no cpython expert, but I think signature dependent semantics for __setitem__, as well as for __getitem__ and __delitem__, could be implemented. And that change alone could be made INDEPENDENT of supporting kwd args, or KeyObjects, or anything else. Furthermore, after signature dependent semantics were implemented (or perhaps at the same time), semantics could easily be extended so that code like this: d[1, b=2] = foo ...on a an object with a __setitem__ signature like this: def __setitem__(self, value, a, b): ... ...gets called like this: d.__setitem__(foo, 1, b=2) BUT, it could also be implemented such that if that same item setting code were made against a signature like this: def __setitem__(self, a, value): ... You would get a TypeError, because the semantic meaning of the signature with just two positional arguments, key_tuple and value, does not support kwd arguments. DOWNSIDE The biggest downside to this that I see is that it would be confusing to the uninitiated, especially in the following case. If the signature dependent semantics proposal were to go forward, and one wrote a signature with two arguments, the following semantic meaning could be mistakenly expected: # mistaken perception of __setitem__ method semantics SIGNATURE === MISTAKEN SEMANTICS (self, a, b) === (self, value, atomic_key) The mistaken belief above stated outright is that the RHS atomic_key is NOT a tuple, it is just b from the LHS. But this would not be correct. The actual semantics, in order to maintain backward compatibility, would not change from what is currently true: # backward compatible semantics SIGNATURE === CORRECT, CURRENT SEMANTICS (self, a, b) === (self, key_tuple, value) The correct understanding illustrated above is that the RHS key_tuple is a tuple (containing b from the LHS) of the form (b,). That seems like potentially a big downside. But maybe not? I don't know. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

On Mon, Aug 24, 2020 at 1:52 PM Christopher Barker wrote:

I’m not at all sure this Idea is possible,

But even if so, there’s a real trick here. The [] operator is not a function call, it is a special operator that “takes” a single expression.

Thing[a, b] is not “getting” two objects, it is getting a single tuple, which is created by the comma. That is, expression:

a,b

Has the same value as:

(a, b)

Or

tuple(a,b)

Which means:

t = tuple(a, b) thing[a, b]

Is exactly the same as

thing[a,b]

And that equivalency needs to be maintained.

In practice, two common use cases treat the resulting tuple essentially semantically differently:

1) Using tuples as dict keys i.e. a single value that happens to be a tuple is a common practice.

2) numpy uses the elements of a tuple as separate indices.

I don’t think the interpreter would have any way to know which of these is intended.

-CHB

The interpreter wouldn't. I'm talking about adding this knowledge of signature dependent semantics to `type`. To implement this, under the hood `type` would detect the signatures with different semantics, and choose to wrap the functions with those signatures in a closure based on the intended semantic meaning. Then everything proceeds as it does today. All of this is possible today, of course, using a metaclass, or using a regular class and the __init_subclass__ method, or using decorators. But my suggestion is to roll it into type. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

On Mon, Aug 24, 2020 at 11:10 AM Ricky Teachey wrote:

\The interpreter wouldn't. I'm talking about adding this knowledge of signature dependent semantics to `type`.

To implement this, under the hood `type` would detect the signatures with different semantics, and choose to wrap the functions with those signatures in a closure based on the intended semantic meaning. Then everything proceeds as it does today.

but it would still not know that: t = (1,2,3) something[t] is the same as: something[1,2,3] would it?

All of this is possible today, of course, using a metaclass, or using a regular class and the __init_subclass__ method, or using decorators. But my suggestion is to roll it into type.

--- Ricky.

"I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

There's another option (but I am not endorsing it): a[1:2, 2, j=4:7, k=3] means: a.__getitem__((slice(1, 2, None), 2, named("j", slice(4, 7, None)), named("k", 3)})) Where named is an object kind like slice, and it evaluates to the pure value, but also has a .name like slice() has .start. In any case, I think that Ricky Teachey might be onto something. Imagine we start from zero. There's no __getitem__. How would you envision it to work? The answer is probably going to be "like a function". that is, if you pass a[1, 2] it will receive two arguments as if it were __new_getitem__(self, v1, v2), with only a couple of magic stuff for slices and Ellipsis. From this, everything would behave rather naturally: a[v1=1, v2=2] would be automatic. and so the flipped one a[v2=2, v1=1] would return the same value. Now, my question is, would it _really_ be a major issue to introduce this __new_getitem__ to be used if available, like we have done already in the past for similar enhanced protocols? Because to me, this solution seems the most rational, flexible, and clear compared to any other option, which in the end are mostly workarounds fraught with either manual work or odd syntax. It would solve the ambiguity of the API, it would allow for the parser to handle the assignment of named arguments to the correct arguments, with the only problem of potentially allowing a[] as valid syntax if __new_getitem__(self, *args, **kwargs) is used. On Sat, 8 Aug 2020 at 03:49, Steven D'Aprano wrote:

On Fri, Aug 07, 2020 at 12:09:28PM -0400, Ricky Teachey wrote:

...

I was actually trying to help the kwd arg case here. As illustrated by the quote I included from Greg Ewing, there seems to be not even close to a consensus over what the semantic meaning of this should be:

m[1, 2, a=3, b=2]

This is Python-Ideas. If we asked for a consensus on what the print function should do, I'm sure we would find at least one person seriously insist that it ought to erase your hard drive *wink*

But seriously, getting consensus is difficult, especially when people seem to be unwilling or unable to articulate why they prefer one behaviour over another, or the advantages vs disadvantages of a proposal.

...

Which could be made to mean one of the following things, or another thing I haven't considered:

1. m.__get__((1, 2), a=3, b=4) # handling of positional arguments unchanged from current behavior

By the way, I assume you meant `__getitem__` in each of your examples, since `__get__` is part of the descriptor protocol.

Advantages:

(1) Existing positional only subscripting does not change (backwards compatible).

(2) Easy to handle keyword arguments.

(3) Those who want to bundle all their keywords into a single object can just define a single `**kw` parameter.

(4) Probably requires little special handling in the interpreter?

(5) Probably requires the minimum amount of implementation effort?

(6) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing.

Disadvantages: none that I can see. (Assuming we agree that this is a useful feature.)

...

2. m.__get__(1, 2, a=3, b=4) # change positional argument handling from current behavior

Advantages:

1. Consistency with other methods and functions.

Disadvantages:

1. Breaks backwards compatibility.

2. Will require a long and painful transition period during which time libraries will have to somehow support both calling conventions.

...

3. m.__get__((1, 2), {'a': 3, 'b': 4}) # handling of positional arguments unchanged from current behavior

I assume that if there are no keyword arguments given, only the first argument is passed to the method (as opposed to passing an empty dict). If not, the advantages listed below disappear.

Advantages:

(1) Existing positional only subscripting does not change (backwards compatible).

(2) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing.

Disadvantages:

(1) Forces people to do their own parsing of keyword arguments to local variables inside the method, instead of allowing the interpreter to do it.

(2) Compounds the "Special case breaks the rules" of subscript methods to keyword arguments as well as positional arguments.

(3) It's not really clear to me that anyone actually wants this, apart from just suggesting it as an option. What's the concrete use-case for this?

...

4. m.__get__(KeyObject( (1, 2), {'a': 3, 'b': 4} )) # change positional argument handling from current behavior only in the case that kwd args are provided

Use-case: you want to wrap an arbitrary number of positional arguments, plus an arbitrary set of keyword arguments, into a single hashable "key object", for some unstated reason, and be able to store that key object into a dict.

Advantage (double-edged, possible):

(1) Requires no change to the method signature to support keyword parameters (whether you want them or not, you will get them).

Disadvantages:

(1) If you don't want keyword parameters in your subscript methods, you can't just *do nothing* and have them be a TypeError, you have to explicitly check for a KeyObject argument and raise:

def __getitem__(self, index): if isinstance(item, KeyObject): raise TypeError('MyClass index takes no keyword arguments')

(2) Seems to be a completely artificial and useless use-case to me. If there is a concrete use-case for this, either I have missed it, (in which case my apologies) or Jonathan seems to be unwilling or unable to give it. But if you really wanted it, you could get it with this signature and a single line in the body:

def __getitem__(self, *args, **kw): key = KeyObject(*args, **kw)

(3) Forces those who want named keyword parameters to parse them from the KeyObject value themselves.

Since named keyword parameters are surely going to be the most common use-case (just as they are for other functions), this makes the common case difficult and the rare and unusual case easy.

(4) KeyObject doesn't exist. We would need a new builtin type to support this, as well as the new syntax. This increases the complexity and maintenance burden of this new feature.

(5) Compounds the "kind of screwy" (Greg's words) nature of subscripting by extending it to keyword arguments as well as positional arguments.

-- Steven _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DAND7B... Code of Conduct: http://python.org/psf/codeofconduct/

-- Kind regards, Stefano Borini

On Tue, Aug 25, 2020 at 4:26 PM Stefano Borini wrote:

In any case, I think that Ricky Teachey might be onto something.

Well when it comes to python ideas, that actually might be a first for me. ;)

Imagine we start from zero. There's no __getitem__. How would you envision it to work?

The answer is probably going to be "like a function". that is, if you pass

a[1, 2]

it will receive two arguments as if it were __new_getitem__(self, v1, v2), with only a couple of magic stuff for slices and Ellipsis. From this, everything would behave rather naturally: a[v1=1, v2=2] would be automatic. and so the flipped one a[v2=2, v1=1] would return the same value.

Actually I think this *may not* be true. Consider, if we were starting at zero ,and we agreed we wanted dict literal behavior to work like the following:

...

...

idx1 = 1,2,3 idx2 = 1,2 idx3 = 1 d = {idx1: "foo", idx2: "bar", idx2: "baz"}

This is what we have today, and I think most agree this is all a Good Thing. Then, we might reason, if I want to be able to do a lookup using tuple literals, I'd do this, just as we do today:

...

...

d[1,2,3] 'foo' d[1,2] 'bar' d[1] 'baz'

No tuple brackets required. I think it is REALLY NICE not to have to do THIS, which is what we'd have to do if things behaved the function-call-way inside square brackets:

...

...

d[(1,2,3)] 'foo' d[(1,2)] 'bar' d[1] 'baz'

Of course all of these DO work today-- it just isn't required to use the tuple brackets. They're optional. Could it be that the convenience of not needing to give the tuple brackets, if we were starting at zero, might win out over the function-call syntax even today? I don't know. I wonder. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler

Now, my question is, would it _really_ be a major issue to introduce this __new_getitem__ to be used if available, like we have done already in the past for similar enhanced protocols? Because to me, this solution seems the most rational, flexible, and clear compared to any other option, which in the end are mostly workarounds fraught with either manual work or odd syntax. It would solve the ambiguity of the API, it would allow for the parser to handle the assignment of named arguments to the correct arguments, with the only problem of potentially allowing a[] as valid syntax if __new_getitem__(self, *args, **kwargs) is used.

On Sat, 8 Aug 2020 at 03:49, Steven D'Aprano wrote:

...

On Fri, Aug 07, 2020 at 12:09:28PM -0400, Ricky Teachey wrote:

...

I was actually trying to help the kwd arg case here. As illustrated by

...

...

quote I included from Greg Ewing, there seems to be not even close to a consensus over what the semantic meaning of this should be:

m[1, 2, a=3, b=2]

This is Python-Ideas. If we asked for a consensus on what the print function should do, I'm sure we would find at least one person seriously insist that it ought to erase your hard drive *wink*

But seriously, getting consensus is difficult, especially when people seem to be unwilling or unable to articulate why they prefer one behaviour over another, or the advantages vs disadvantages of a proposal.

...

Which could be made to mean one of the following things, or another

...

...

haven't considered:

1. m.__get__((1, 2), a=3, b=4) # handling of positional arguments unchanged from current behavior

By the way, I assume you meant `__getitem__` in each of your examples, since `__get__` is part of the descriptor protocol.

Advantages:

(1) Existing positional only subscripting does not change (backwards compatible).

(2) Easy to handle keyword arguments.

(3) Those who want to bundle all their keywords into a single object can just define a single `**kw` parameter.

(4) Probably requires little special handling in the interpreter?

(5) Probably requires the minimum amount of implementation effort?

(6) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing.

Disadvantages: none that I can see. (Assuming we agree that this is a useful feature.)

...

2. m.__get__(1, 2, a=3, b=4) # change positional argument handling from current behavior

Advantages:

1. Consistency with other methods and functions.

Disadvantages:

1. Breaks backwards compatibility.

2. Will require a long and painful transition period during which time libraries will have to somehow support both calling conventions.

...

3. m.__get__((1, 2), {'a': 3, 'b': 4}) # handling of positional arguments unchanged from current behavior

I assume that if there are no keyword arguments given, only the first argument is passed to the method (as opposed to passing an empty dict). If not, the advantages listed below disappear.

Advantages:

(1) Existing positional only subscripting does not change (backwards compatible).

(2) Requires no extra effort for developers who don't need or want keyword parameters in their subscript methods. Just do nothing.

Disadvantages:

(1) Forces people to do their own parsing of keyword arguments to local variables inside the method, instead of allowing the interpreter to do it.

(2) Compounds the "Special case breaks the rules" of subscript methods to keyword arguments as well as positional arguments.

(3) It's not really clear to me that anyone actually wants this, apart from just suggesting it as an option. What's the concrete use-case for this?

...

4. m.__get__(KeyObject( (1, 2), {'a': 3, 'b': 4} )) # change positional argument handling from current behavior only in the case

the thing I that

...

...

kwd args are provided

Use-case: you want to wrap an arbitrary number of positional arguments, plus an arbitrary set of keyword arguments, into a single hashable "key object", for some unstated reason, and be able to store that key object into a dict.

Advantage (double-edged, possible):

(1) Requires no change to the method signature to support keyword parameters (whether you want them or not, you will get them).

Disadvantages:

(1) If you don't want keyword parameters in your subscript methods, you can't just *do nothing* and have them be a TypeError, you have to explicitly check for a KeyObject argument and raise:

def __getitem__(self, index): if isinstance(item, KeyObject): raise TypeError('MyClass index takes no keyword arguments')

(2) Seems to be a completely artificial and useless use-case to me. If there is a concrete use-case for this, either I have missed it, (in which case my apologies) or Jonathan seems to be unwilling or unable to give it. But if you really wanted it, you could get it with this signature and a single line in the body:

def __getitem__(self, *args, **kw): key = KeyObject(*args, **kw)

(3) Forces those who want named keyword parameters to parse them from the KeyObject value themselves.

Since named keyword parameters are surely going to be the most common use-case (just as they are for other functions), this makes the common case difficult and the rare and unusual case easy.

(4) KeyObject doesn't exist. We would need a new builtin type to support this, as well as the new syntax. This increases the complexity and maintenance burden of this new feature.

(5) Compounds the "kind of screwy" (Greg's words) nature of subscripting by extending it to keyword arguments as well as positional arguments.

-- Steven _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DAND7B... Code of Conduct: http://python.org/psf/codeofconduct/

-- Kind regards,

Stefano Borini _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FK3LOX... Code of Conduct: http://python.org/psf/codeofconduct/

On 26/08/20 10:03 am, Stefano Borini wrote:

But you have a point that whatever the implementation might be, it has to play nice with the current dict() behavior. Yet, if we were to add an enhanced dunder, nothing for the current dict would change.

Despite arguing against it earlier, I think I'm starting to shift towards being in favour of a new dunder. But only if it can be justified as existing in its own right alongside __getitem__, without any intention to deprecate or remove the latter. I think an argument can be made that the new dunder -- let's call it __getindex__ for now -- shouldn't be considered part of the mapping protocol. It's a new thing for classes that want to use the indexing notation in ways that don't fit the simple idea of mapping a key object to a value object. The only drawback I can think of right now is that it would make possible objects for which x[1, 2] and x[(1, 2)] were not equivalent. I don't think that's a serious problem. A class would always be able to arrange for its __getindex__ method to *make* them equivalent, and things like pandas would be encouraged to do so. For things like generic types, which use the index notation for something that isn't really indexing at all, it probably doesn't matter much. -- Greg

On Tue, Aug 25, 2020, 7:35 PM Greg Ewing wrote:

On 26/08/20 10:03 am, Stefano Borini wrote:

...

But you have a point that whatever the implementation might be, it has to play nice with the current dict() behavior. Yet, if we were to add an enhanced dunder, nothing for the current dict would change.

Despite arguing against it earlier, I think I'm starting to shift towards being in favour of a new dunder. But only if it can be justified as existing in its own right alongside __getitem__, without any intention to deprecate or remove the latter.

...

The only drawback I can think of right now is that it would make possible objects for which x[1, 2] and x[(1, 2)] were not equivalent. I don't think that's a serious problem. A class would always be able to arrange for its __getindex__ method to *make* them equivalent, and things like pandas would be encouraged to do so. For things like generic types, which use the index notation for something that isn't really indexing at all, it probably doesn't matter much.

-- Greg

To be backwards compatible, any proposed change has to pass this test: ############ """Test #1""" class D: def __getitem__(self, key): return key d = D() # note the commas assert d[1,] == f((1,)) ################ Where f is some function that mimicks how python "translates" the "stuff" inside the square brackets. It is something like this, not quite: def f(*args): if not args: # empty brackets not allowed raise SyntaxError() if len(args) ==1: return args[0] return args The desire of many (me included) seems to be to make it possible for the subscript operator to utilize syntax very similar, or identical, to the behavior of a function call. In other words, to make it possible to write some class, Q, that allows the following tests to pass: ################# """Test #2""" # note the commas assert q[1,] == f(1,) assert q[1] == f(1) assert q[(1,)] == f((1,)) ################## The problem is, I don't think it is possible to change python internals in a backwards compatible way so that we can write a function that passes all of these tests. Because the index operator recognizes this as a tuple: 1, But function calls do not. This is an impasse. Does this constitute a big problem for the idea of providing new dunders? I'm not sure.

On Tue, Aug 25, 2020 at 4:41 PM Stefano Borini wrote:

On Sat, 8 Aug 2020 at 02:34, Stephan Hoyer wrote:

...

I'm sorry, I did a poor job of editing this.

To fill in my missing word: From my perspective, the *only* reasonable way to add keyword arguments to indexing would be in a completely backwards compatible way with **kwargs.

Let me clarify that I don't like the kwargs solution to the current getitem, and the reason is simple.

The original motivation for PEP-472 was to do axis naming, that is, to be sure you would be able to clearly indicate (if you so wished) data[23, 2] explicitly as e.g. data[day=23, detector=2] or data[detector=2, day=23]

But that assumes that all keyword axes can be mapped to positional axes. This is not the case in xarray, there are axes that are keyword-only. And it precludes other uses-cases that have been mentioned, such as parameters for indexing. So I understand that this was the original motivation, I don't think it is good to restrict to it only this use-case when there are other valid use-cases.

If you have **kwargs, now the first case would send the two values to the nameless index, the second case to the kwargs, and inside the getitem you would have to reconcile the two, especially if someone then writes data[24, 5, day=23, detector=2] typing of course has the same issue. What this extended syntax is supposed to be used for is to define specialisation of generics. but it's difficult to now handle the cases List[Int] vs List[T=Int] vs List[Int, T=int], which are really telling the same thing the first two, and something wrong (TypeError multiple arguments) the second.

That is a problem that anyone using **kwargs in functions has to deal with. It is neither a new problem nor a unique one, and there are plenty of ways to deal with it.

On Wed, Aug 26, 2020 at 11:31:25AM +1200, Greg Ewing wrote:

I think an argument can be made that the new dunder -- let's call it __getindex__ for now -- shouldn't be considered part of the mapping protocol. It's a new thing for classes that want to use the indexing notation in ways that don't fit the simple idea of mapping a key object to a value object.

Most existing uses of subscripts already don't fit that key:value mapping idea, starting with lists and tuples. Given `obj[spam]`, how does the interpreter know whether to call `__getitem__` or `__getindex__`? What if the class defines both?

The only drawback I can think of right now is that it would make possible objects for which x[1, 2] and x[(1, 2)] were not equivalent.

Right now, both sets of syntax mean the same thing and call the same method, so you are introducing a backwards incompatible change that will break code. Can we postpone this proposal for Python 6000 in 2050? -- Steve

On 26/08/20 1:59 pm, Steven D'Aprano wrote:

Most existing uses of subscripts already don't fit that key:value mapping idea, starting with lists and tuples.

Not sure what you mean by that.

Given `obj[spam]`, how does the interpreter know whether to call `__getitem__` or `__getindex__`? What if the class defines both?

If it has a __getindex__ method, it calls that using normal function parameter passing rules. Otherwise it uses a fallback something like def __getindex__(self, *args, **kwds): if kwds: raise TypeError("Object does not support keyword indexes") if not args: raise TypeError("Object does not accept empty indexes") if len(args) == 1: args = args[0] return self.__getitem__(args)

Right now, both sets of syntax mean the same thing and call the same method, so you are introducing a backwards incompatible change that will break code.

No existing object will have a __getindex__ method[1], so it won't change any existing behaviour. [1] Or at least not one that we aren't entitled to break. -- Greg

On Tue, Aug 25, 2020 at 10:51:42PM -0400, Ricky Teachey wrote:

...

The only way it could tell that would be to inspect *at runtime* the `__setitem__` method. And it would have to do this on every subscript call. Introspection is likely to be slow, possibly very slow. This would make subscripting slow.

Well it would not have to inspect the signature on every subscript call,

Python is a very dynamic language and objects can change their own methods and even their class at any time, so, yes, it will have to inspect the signature on every call. Otherwise you are changing the language execution model. Demonstration: py> class Dynamic: ... def __getitem__(self, arg): ... print("original") ... type(self).__getitem__ = lambda *args: print("replaced") ... py> py> x = Dynamic() py> x[0] original py> x[0] replaced [...]

People have actually been asking for ways to make subscripting operator act more like a function call, so that's not true.

Yes, people have asked for keyword arguments. This proposal doesn't get us any closer to the feature wanted.

And it could be useful.

It doesn't give subscripting any additional functionality that doesn't already exist. It's a purely cosmetic change.

And it does help address the problem of incongruity (though not perfectly) between the way a function call handles args and kwargs, and the way the subscript operator does it. And it is totally backwards compatible except for the case [...]

So not actually totally backwards compatible. Please stop calling things "totally backwards compatible" if they aren't totally backwards compatible. If you change the behaviour of existing legal code, it's not backwards compatible.

...

Maybe we wouldn't have designed subscripting this way back in Python 1 if we know what we know now, but it works well enough, and we have heard from numpy developers like Stephan Hoyer that this is not a problem that needs fixing. Can we please stop trying to "fix" positional subscripts?

I'm not done trying, sorry. I think the incongruity is a problem.

A problem for whom? A problem in what way? PEP 472 goes into detail about the sorts of things people find really hard to do with subscripting because of the lack of keywords. Where is the discussion by people about the things that are really hard to do because of the comma handling in subscripts? [...]

Remember: the signature dependent semantics proposal is to maintain backwards compatibility, 100%, for any code that has been following the very clear intent of the item dunder methods.

The Python language doesn't follow the rule "Do what we intend you to do", it follows the rule "this is how the language works, do whatever the language allows". PEP 5 doesn't mention the word "intent" and doesn't say "its okay to break people's code if they are using Python in ways we don't like". You don't get to say that breaking legal code that works today is okay because it doesn't follow "the intent". It's still a breaking change and you have to convince the Steering Council that breaking other people's code for the sake of fixing an incongruity is worthwhile. Imagine it was code *you* relied on that broke, and you no longer had the source code for it, it was a library compiled away in a .pyc file and the company that distributed it has gone broke and the source code lost, and replacing that library is going to cost your business a year of time and a hundred thousand dollars in development costs. But that's okay, because an incongruity that makes no difference to anyone's code has been fixed. Now maybe you can convince the Steering Council that this is a clear win for the broader community. We do break backwards compatibility, sometimes, if the risks are small enough and the benefits large enough. But be honest about what you are doing.

Sure, it might break some code because somebody, somewhere, has a __getitem__ method written like this:

def __getitem__(self, a, b=None, c=None): ... [...] But are you really saying there is a very important responsibility not to break that person's code?

Yes. -- Steve

On 27/08/20 12:53 am, Steven D'Aprano wrote:

Presumably the below method is provided by `object`. If not, what provides it?

...

def __getindex__(self, *args, **kwds): if kwds: raise TypeError("Object does not support keyword indexes") if not args: raise TypeError("Object does not accept empty indexes")

It's not literally a method, I just wrote it like that to illustrate the semantics. It would be done by the interpreter as part of the process of translating indexing operations into dunder calls.

What is your reasoning behind prohibiting keywords, when we're right in the middle of a discussion over PEP 474 which aims to allow keywords?

We're falling back to __getitem__ here, which doesn't currently allow keywords, and would stay that way. The point of this proposal is to not change __getitem__. If you want to get keywords, you provide __getindex__.

...

if len(args) == 1: args = args[0] return self.__getitem__(args)

This is going to slow down the most common cases of subscripting: the interpreter has to follow the entire MRO to find `__getindex__` in object, which then dispatches to the `__getitem__` method.

No, it would be done by checking type slots, no MRO search involved.

In your earlier statement, you said that it would be possible for subscripting to mean something different depending on whether the comma-separated subscripts had parentheses around them or not:

obj[(2, 3)] obj[2, 3]

How does that happen?

If the object has a __getindex__ method, it gets whatever is between the [] the same way as a normal function call, so comma-separated expressions become separate positional arguments. -- Greg

On 27/08/20 3:51 am, Ricky Teachey wrote:

On Wed, Aug 26, 2020 at 11:30 AM Greg Ewing mailto:greg.ewing@canterbury.ac.nz> wrote:

No, it would be done by checking type slots, no MRO search involved.

Can you elaborate on this for my understanding?

For frequently-used special methods such as __getitem__, the type object contains direct pointers to C functions (so-called "type slots"). These are set up when the type object is created (and updated if the MRO is modified). For classes written in Python, they are filled with wrappers that invoke the user's Python code. So the implementation of the bytecodes that perform indexing would first look for a value in the slot for __getindex__; if it finds one, it calls it like a normal function. Otherwise it does the same as now with the slot for __getitem__. The overhead for this would be just one extra C pointer check, which you would be hard-pressed to measure.

the index operator recognizes this as a tuple:

1,

But function calls do not.

We could probably cope with that by generating different bytecode when there is a single argument with a trailing comma, so that a runtime decision can be made as to whether to tupleify it. However, I'm not sure whether it's necessary to go that far. The important thing isn't to make the indexing syntax exactly match function call syntax, it's to pass multiple indexes as positional arguments to __getindex__. So I'd be fine with having to write a[(1,)] to get a one-element tuple in both the old and new cases. It might actually be better that way, because having trailing commas mean different things depending on the type of object being indexed could be quite confusing. -- Greg

On Thu, Aug 27, 2020 at 03:28:07AM +1200, Greg Ewing wrote:

On 27/08/20 12:53 am, Steven D'Aprano wrote:

...

Presumably the below method is provided by `object`. If not, what provides it?

...

def __getindex__(self, *args, **kwds): if kwds: raise TypeError("Object does not support keyword indexes") if not args: raise TypeError("Object does not accept empty indexes")

It's not literally a method, I just wrote it like that to illustrate the semantics. It would be done by the interpreter as part of the process of translating indexing operations into dunder calls.

Okay, so similar to my suggestion that this would be better implemented in the byte-code rather than as a method of object.

...

What is your reasoning behind prohibiting keywords, when we're right in the middle of a discussion over PEP 474 which aims to allow keywords?

We're falling back to __getitem__ here, which doesn't currently allow keywords, and would stay that way. The point of this proposal is to not change __getitem__. If you want to get keywords, you provide __getindex__.

Point of order: **the getitem dunder** already allows keywords, and always has, and always will. It's just a method. It's the **subscript (pseudo-)operator** which doesn't support keywords. This is a syntax limitation, not a limitation of the dunder method. If the interpreter supports the syntax, it's neither here nor there to the interpreter whether it calls `__getitem__` or `__getindex__` or `__my_hovercraft_is_full_of_eels__` for that matter. So if you want to accept keywords, you just add keywords to your existing dunder method. If you don't want them, don't add them. We don't need a new dunder just for the sake of keywords.

...

This is going to slow down the most common cases of subscripting: the interpreter has to follow the entire MRO to find `__getindex__` in object, which then dispatches to the `__getitem__` method.

No, it would be done by checking type slots, no MRO search involved.

Okay, I didn't think of type slots. But type slots are expensive in other ways. Every new type slot increases the size of objects, and I've seen proposals for new dunders knocked back for that reason, so presumably the people care about the C-level care about the increase in memory and complexity from adding new type slots. Looking here: https://docs.python.org/3/c-api/typeobj.html I see that `__setitem__` and `__delitem__` are handled by the same type slot, so presumably the triplet get-, set- and del- index type slots would be the same. Still, that means adding two new type slots to both sequence and mapping objects. (I assume that it's not *all* objects that carry these slots. If it is all objects, that makes the cost of this proposal correspondingly higher.) So if I understand it correctly, we have some choices when it comes to sequence/mapping types: 1. the existing `__*item__` methods keep their slots, and new `__*index__` slots are created, which makes both the item and index dunders fast but increases the size of every object which uses any of those methods; 2. the existing item slots stay as they are, there are no new index slots, which keeps objects the same size but the new index protocol will be slow; 3. the existing item slots are repurposed for index, which keeps objects the same size, and the new protocol fast, but makes calling item dunders slow; 4. and just for completion because of course this is not going to happen, we could remove the existing item slots and not add index slots so that both protocols are equally slow; 5. alternatively, we could leave the existing C-level sequence and mapping objects alone, and create *four* brand new C-level objects: - a sequence object that supports only the new index protocol; - a sequence object that supports both index and item protocols; - and likewise two new mapping objects. Do I understand this correctly? Have I missed any options? Assuming I do, 4 is never going to happen and each of the others have some fairly large disadvantages and costs in speed, memory, and complexity. Without a correspondingly large advantage to this new `__*index__` protocol, I don't see this going anywhere.

...

In your earlier statement, you said that it would be possible for subscripting to mean something different depending on whether the comma-separated subscripts had parentheses around them or not:

obj[(2, 3)] obj[2, 3]

How does that happen?

If the object has a __getindex__ method, it gets whatever is between the [] the same way as a normal function call, so comma-separated expressions become separate positional arguments.

The compiler doesn't know whether the object has the `__getindex__` method at compile time, so any process that relies on that knowledge isn't going to work. There can only be one set of parsing rules that applies regardless of whether the object defines the item dunders or the index dunders or neither. Right now, if you call `obj[1,]` the dunder receives the tuple (1,) as index. If it were treated as function call syntax, that would receive a single argument 1 instead. It were treated as a tuple, as required by backwards compatibility, that's an inconsistency between subscripts and function calls, and the whole point of your proposal is to remove that inconsistency. Rock (you are here) Hard Place. Do you break existing code, or fail in your effort to remove the inconsistencies? I don't care two hoots about the inconsistencies, I just want to use keywords in my subscripts, so for me the answer is obvious: keep backwards compatibility, and there is no need to add new dunders to only partially fix something which isn't a problem. Another inconsistency: function call syntax looks like this: call ::= primary "(" [argument_list [","] | comprehension] ")" which means we can write generator comprehensions inside function calls without additional parentheses: func(expr for x in items) # unambiguously a generator comprehension This is nice because the round brackets of the function call match the round brackets used in generator comprehensions, so it is perfctly consistent and unambiguous. But if you do that in a subscript, we currently get a syntax error. If we allowed it, it would be pretty weird for the square brackets of the subscript to create a *generator* comprehension rather than a list comprehension. But we surely don't want a list comprehension by default: obj[(expr for x in items)] # unambiguously a generator comprehension obj[[expr for x in items]] # unambiguously a list comprehension obj[expr for x in items] # and this is... what? It looks like it should be a list comprehension (it has square brackets, right?) but we probably don't want it to be a list comp, we'd prefer it to be a generator comp because they are more flexible. Only that would look weird and would lead to all sorts of questions about why list comprehension syntax sometimes gives a list and sometimes a generator. But if we leave it out, we have an inconsistency between subscripting and function calls, and for those who are motivated by removing that inconsistency, that's a Bad Thing. For me, again, the answer is obvious: we don't have to support this for the sake of consistency, because consistency isn't the motivation. I just want keywords. -- Steve

On Thu, Aug 27, 2020, 00:56 Bruce Leban wrote:

A bunch of the conversation here is how to handle both positional and keyword arguments with a single signature. Let me suggest an alternative. At compile time, we know if the call is made with keyword arguments or not.

a[1] positional only a[b=1] keyword only a[1, b=1] both a[**kwargs] keyword only

I suggest that in the first case it calls __getitem__(self, args) as it does now and in the other cases it calls __getitemx__(self, args, **kwargs) instead (the signature is what matters here, don't bikeshed the name). Some people have proposed arbitrary signatures for __getitemx__ but I think that's an unnecessary degree of complexity for little benefit. Personally, I'm not even sure I'd ever want to mix args and kwargs but including that in the signature preserves that possibility for others that find it useful.

To explain further: when I use [...] without positional arguments the code works exactly as it does today with all args in a tuple. Therefore, for consistency when I add a keyword argument that should not change the args value which is why both signatures include a single args parameter.

If you write a form that uses a keyword argument, and __getitemx__ does not exist then it would raise an error other than KeyError (either TypeError or AttributeError, with the latter requiring no special handling).

--- Bruce

What is the advantage of this over adding keyword arguments to the existing find dunders?

On Thu, Aug 27, 2020 at 11:13:38PM +1200, Greg Ewing wrote:

...

So if you want to accept keywords, you just add keywords to your existing dunder method. If you don't want them, don't add them. We don't need a new dunder just for the sake of keywords.

Nobody disputes that it *could* be made to work that way. But I'm not convinced that it's the *best* way for it to work. The killer argument in my mind is what you would have to do to make an object where all of the following are equivalent:

a[17, 42] a[time = 17, money = 42] a[money = 42, time = 17]

I don't think that something like that is exactly the intended use-case for the feature, at least not intended by *me*, that looks more like it should be a function API. But I'm sure people will want to do it, so we should allow it even if it is an (ever-so-slight) abuse of subscript notation. (But I note that one of the PEP co-authors, Joseph Martinot-Lagarde, has now suggested that we bypass this question by simply requiring an index, always.)

With a fresh new dunder, it's dead simple:

def __getindex__(self, time, money): ...

Okay, that's an advantage if you're writing this sort of code. I'm not sure its a big enough advantage to require such changes, new dunders, new byte codes, etc, but YMMV. [...]

...

But type slots are expensive in other ways. Every new type slot increases the size of objects, and I've seen proposals for new dunders knocked back for that reason, so presumably the people care about the C-level care about the increase in memory and complexity from adding new type slots.

It doesn't seem like a serious problem to me. Type objects are typically created once at program startup, and they're already quite big. If it's really a concern, the new slots could be put in a substructure, so the type object would only be bigger by one pointer if they weren't being used.

*shrug* I don't know enough about type slots, except I am confident that I have seen dunders rejected because it would require adding new type slots. But maybe I have the details wrong and am thinking of something else.

Another possibility would be to have them share the slots for the existing methods, with flags indicating which variant is being used for each one. A given type is only going to need either __getindex__ or __getitem__, etc., not both.

In which case, why do we need new names for the methods? Effectively they're the same method, just with different behaviour and a different name, but since you can only use one even if you define both, why would you define both? I would expect it is less confusing and error prone to just say "Starting in version 3.10 subscript parsing changes and item dunders should change their signature ..." than to have two sets of dunders, especially when the names are so similar that invariably people will typo the names: class X: def __getindex__(self, ...): # later def __setitem__(self, ...): # oops I'd rather a conditional: class X: if sys.version_info >= (3, 10): # New signatures def __getitem__(self, ...): else: # Old signatures Or conditionally inherit the dunders from a mixin: if sys.version_info >= (3, 10): from this_is_the_future import item_mixin: else: from legacy import item_mixin class X(item_mixin): pass Having to remember which name goes with which version is going to be annoying and tricky, especially since for many simple cases like lists and dicts the getitem and getindex versions will be identical. [...]

...

5. alternatively, we could leave the existing C-level sequence and mapping objects alone, and create *four* brand new C-level objects:

- a sequence object that supports only the new index protocol; - a sequence object that supports both index and item protocols; - and likewise two new mapping objects.

I don't follow this one. How can there be both old and new protocols without adding new dunder methods?

I didn't say anything about inventing new protocols without new dunders. All five of my alternatives were under the assumption that the new protocol and new dunders was going ahead. But as I mention above, if we did introduce this, I'd rather we keep the dunders and change the name rather than introduce confusable names. -- Steve

On Sun, 30 Aug 2020 at 04:09, Steven D'Aprano wrote:

...

Nobody disputes that it *could* be made to work that way. But I'm not convinced that it's the *best* way for it to work. The killer argument in my mind is what you would have to do to make an object where all of the following are equivalent:

a[17, 42] a[time = 17, money = 42] a[money = 42, time = 17]

I don't think that something like that is exactly the intended use-case for the feature, at least not intended by *me*, that looks more like it should be a function API.

The above feature is well intended and part of the original PEP, mostly for two reasons: 1. During my career I encountered issues where people confused the axis of a matrix. This is especially problematic when the matrix is symmetric. 2. pandas has this exact problem with column names, and while they have a workaround, in my opinion it is suboptimal

But as I mention above, if we did introduce this, I'd rather we keep the dunders and change the name rather than introduce confusable names.

that is something it's hard to fight against. unless we call it something like _extended_getitem_? getItemEx was proposed by GvR for the C API... it seems to be a frequent convention.

On Sun, Aug 30, 2020 at 05:49:50PM +1200, Greg Ewing wrote:

On 30/08/20 3:06 pm, Steven D'Aprano wrote:

...

On Thu, Aug 27, 2020 at 11:13:38PM +1200, Greg Ewing wrote:

...

a[17, 42] a[time = 17, money = 42] a[money = 42, time = 17]

I don't think that something like that is exactly the intended use-case for the feature,

I don't see why we need to pick one use case to bless as the official "true" use case.

We do that for every other operator and operator-like function. Operator overloading is a thing, so if you want `a + b` to mean looking up a database for `a` and writing it to file `b`, you can. But the blessed use-cases for the `+` operator are numeric addition and sequence concatenation. Anything else is an abuse of the operator: permitted, possibly a *mild* abuse, but at best it's a bit wiffy. (There are a few gray areas, such as array/matrix/vector addition, which sneak into the "acceptable" area by virtue of their long usage in mathematics.) If you don't like the term "abuse", I should say I am heavily influenced by its use in mathematics where "abuse of notation" is not considered a bad thing: https://en.wikipedia.org/wiki/Abuse_of_notation In any case, as you say (and agreeing with me) people will want to do something like this, and it is not my intent to stop them.

Plus we have at least one real-world use case, i.e. xarray, where it would be a very obvious thing to do.

Indeed, and I can think of at least two use-cases where I would immediately use it myself, if I could.

...

But I'm sure people will want to do it, so we should allow it even if it is an (ever-so-slight) abuse of subscript notation.

How is it an abuse? To me it seems less of an abuse than other uses where the keyword args *don't* represent index values.

Compares to the majority of realistic examples given in this discussion and the PEP, your example doesn't look to me like an item or key lookup. It looks more like a function call of some sort. I know that, in a sense, subscript lookups are like function calls, and I also acknowledge that I don't know your intended semantics of that example. It might even be that you intended it as a lookup of some kind of record with time=17 and money=42. But let's not get too bogged down into not-picking. We agree that people should be allowed to do this. [...]

...

"Starting in version 3.10 subscript parsing changes and item dunders should change their signature ..."

class X: if sys.version_info >= (3, 10): # New signatures def __getitem__(self, ...): else: # Old signatures

Are you volunteering to rewrite all existing Python code that defines item dunders? :-)

Of course not. This is why we ought to be cautious about introducing backwards-incompatible changes. I don't think that we should care about this "problem" that subscripting only takes a single subscript. But if we do "fix" it, I'd rather the straight-forward future directive (and a long, long deprecation period for the old behaviour) than two sets of confusingly similar getters and setters.

...

Having to remember which name goes with which version is going to be annoying and tricky,

I'm not wedded to the name __getindex__. It can be called __newgetitem__ or __getitemwithfancyparemeterpassing__ if people thought that would be less confusing.

The trouble with naming anything "new" is that in ten years time it's not new any more, and you could end up needing a "new new" method too. -- Steve

A digression on abuse of operators, notation, and "abuse of notation". Steven D'Aprano writes:

On Sun, Aug 30, 2020 at 05:49:50PM +1200, Greg Ewing wrote:

...

I don't see why we need to pick one use case to bless as the official "true" use case.

We do that for every other operator and operator-like function.

Operator overloading is a thing, so if you want `a + b` to mean looking up a database for `a` and writing it to file `b`, you can. But the blessed use-cases for the `+` operator are numeric addition and sequence concatenation.

Of course the example is abusive. But not being "blessed" isn't the reason. (And shouldn't the notation be "b += a" for that operation? ;-)

Anything else is an abuse of the operator: permitted, possibly a *mild* abuse, but at best it's a bit wiffy.

I don't agree. The reasons for picking a "blessed" use case for operators do not include "to exclude other uses". Rather, they are to help the language designer decide things like precedence and associativity, to give developers some guidance in choosing an operator to overload for a two-argument function, and to provide readers with a familiar analog for the behavior of an operator. For example, sequence concatenation is not "blessed because it's blessed". It's blessed because it just works. What does "5 * 'a' + 'b'" mean to the reader? Is the result 'aaaaab', or 'ababababab'? The reader knows it's the former, and that is the right answer for the developer because 'ababababab' == 5 * 'ab' -- there's another way to construct that without parenthesis. Why is 5 * 'a' == 'aaaaa'? Because 5 * 'a' == 'a' + 'a' + 'a' + 'a' + 'a'. None of this is an accident. There's nothing wiffy about it. Recall that some people don't like the use of '+' for sequence concatenation because the convention in abstract algebra is that '+' denotes an commutative associative binary operation, while '*' need not be commutative. But I think you don't want to use '*' for sequence concatenation in Python because 5 * 'a' * 'b' is not associative: it matters which '*' you execute first. Where things get wiffy is when you have conflicting requirements, such as if for some reason you want '*' to bind more tightly than '+' which in turn should bind at least as tightly as '/'. You can swap the semantics of '+' and '/' but that may not "look right" for the types involved. You may want operator notation badly for concise expression of long programs, but it will always be wiffy in this circumstance. And use of "a + b" to denote "looking up a database for `a` and writing it to file `b`" is abusive, but not of notation. It's an abuse because it doesn't help the reader remember the semantics, and I don't think that operation is anywhere near as composable as arithmetic expressions are.

If you don't like the term "abuse", I should say I am heavily influenced by its use in mathematics where "abuse of notation" is not considered a bad thing:

https://en.wikipedia.org/wiki/Abuse_of_notation

I think operator overloading and abuse of notation are quite different things, though. Using '*' to denote matrix multiplication or "vertical composition of morphisms in a category" is not an abuse of notation. It's overloading, using the same symbol for a different operation in a different domain. Something like numpy broadcasting *is* an abuse of notation, at least until you get used to it: you have nonconforming matrices, so the operation "should" be undefined (in the usual treatment of matrices). If you take "undefined" *literally*, however, there's nothing to stop you from *defining* the operation for a broader class of environments, and it's not an abuse to do so. So if you have a class Vector and a function mean(x: Vector) -> Vector, v = Vector(1, 2, 3) m = mean(v) # I Feel Like a Number, but I am not variance = (v - m) * (v - m) / len(v) "looks like" an abuse of notation to the high school student targeted by the "let's have a Matrix class in stdlib" thread. But as Numpy arrays show, that's implementable, it's well-defined, and once you get used to it, the meaning is clear, even in the general case. You could do the same computation with lists v = [1, 2, 3] m = sum(v)/len(v) variance = (v - len(v) * [m]) * (v - len(v) * [m]) / len(v) but I don't think that's anywhere near as nice. To call definitions extending the domain of an operator "abuse of notation" is itself an abuse of language, because abuse of notation *can't happen* in programming! Abuse of notation is synecdoche (use of a part to denote the whole) or metonymy (use of a thing to represent something related but different). But do either in programming, and you get an incorrect program. No matter how much you yell at Guido, the Python translator is never going to "get" synecdoche or metonymy -- it's just going to do what you don't want it to do.

On Wed, Sep 2, 2020 at 3:48 AM Stephen J. Turnbull wrote:

A digression on abuse of operators, notation, and "abuse of notation".

Steven D'Aprano writes:

...

On Sun, Aug 30, 2020 at 05:49:50PM +1200, Greg Ewing wrote:

...

...

I don't see why we need to pick one use case to bless as the official "true" use case.

We do that for every other operator and operator-like function.

Operator overloading is a thing, so if you want `a + b` to mean looking up a database for `a` and writing it to file `b`, you can. But the blessed use-cases for the `+` operator are numeric addition and sequence concatenation.

Of course the example is abusive. But not being "blessed" isn't the reason. (And shouldn't the notation be "b += a" for that operation? ;-)

Possibly a better example would be path division in Python, or stream left/right shift in C++. What does it mean to divide a path by a string? What does it mean to left-shift std::cout by 5? Those uses don't make a lot of sense based on the operations involved, but people accept them because the symbols look right:

...

...

pathlib.Path("/") / "foo" / "bar" / "quux" PosixPath('/foo/bar/quux')

Is that abuse of notation or something else? Whatever it is, it's not "operator overloading" in its normal sense; division is normally the inverse of multiplication, but there's no way you can multiply that by "quux" to undo that last operation, and nobody would expect so. Maybe we need a different term for this kind of overloading, where we're not even TRYING to follow the normal semantics for that operation, but are just doing something because it "feels right". ChrisA

On Tue, Sep 1, 2020 at 4:57 PM Greg Ewing wrote:

On 2/09/20 2:24 am, Steven D'Aprano wrote:

...

On Sun, Aug 30, 2020 at 05:49:50PM +1200, Greg Ewing wrote:

...

On 30/08/20 3:06 pm, Steven D'Aprano wrote:

...

On Thu, Aug 27, 2020 at 11:13:38PM +1200, Greg Ewing wrote:

...

a[17, 42] a[time = 17, money = 42] a[money = 42, time = 17]

Compares to the majority of realistic examples given in this discussion and the PEP, your example doesn't look to me like an item or key lookup. It looks more like a function call of some sort.

I know that, in a sense, subscript lookups are like function calls, and I also acknowledge that I don't know your intended semantics of that example.

It was a reference to earlier discussions of pandas and xarray, where there is a notion of axes having names, and a desire to be able to specify index values by name, for the same reasons that we sometimes want to specify function arguments by name.

It's true that you can't tell just by looking at an indexing expression what the semantics of the keywords are, but the same is true of function calls. We rely on contextual knowledge about what the function does in order to interpret the keywords.

Likewise here. If you know from context that a is an array-like object with axes named 'time' and 'money', then I think the meaning of the indexing expression will be quite clear. I also think that it's something people will naturally expect to be able to use index keywords for -- to the point that if they can't, their reaction will be "Why the flipping heck not?"

Which is why I was somewhat perplexed when it was suggested that we should discount this use case purely because it wasn't cited in the original proposal (which turned out to be wrong anyway).

I agree it's a fine use case. Using the currently prevailing proposal (which I steadfastly will refer to as "Steven's proposal") it's quite possible to implement this. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On Tue, Sep 1, 2020, 9:06 PM Ricky Teachey wrote:

On Tue, Sep 1, 2020, 8:35 PM Guido van Rossum wrote:

...

On Tue, Sep 1, 2020 at 4:57 PM Greg Ewing wrote:

...

On 2/09/20 2:24 am, Steven D'Aprano wrote:

...

On Sun, Aug 30, 2020 at 05:49:50PM +1200, Greg Ewing wrote:

...

On 30/08/20 3:06 pm, Steven D'Aprano wrote:

...

On Thu, Aug 27, 2020 at 11:13:38PM +1200, Greg Ewing wrote:

> a[17, 42] > a[time = 17, money = 42] > a[money = 42, time = 17]

I agree it's a fine use case. Using the currently prevailing proposal (which I steadfastly will refer to as "Steven's proposal") it's quite possible to implement this.

...

-- --Guido van Rossum (python.org/~guido)

Can someone tell me...

Is this right? Wrong? The hard way? The slow way?

Actually just realized I definitely did get it wrong: def __getitem__(self, key=MISSING, time=MISSING, money=MISSING): if time is MISSING or money is MISSING: if time is not MISSING and key is not MISSING: money, key = key, MISSING elif money is not MISSING and key is not MISSING: time, key = key, MISSING elif time is MISSING and money is MISSING: (time, money), key = key, MISSING if key is not MISSING: raise TypeError() Still not sure if that is right.

On 2/09/20 8:35 am, Chris Angelico wrote:

Maybe we need a different term for this kind of overloading, where we're not even TRYING to follow the normal semantics for that operation, but are just doing something because it "feels right".

Operator repurposing? -- Greg

On Tue, Sep 1, 2020, at 21:06, Ricky Teachey wrote:

Here's my attempt, it is probably lacking. I'm five years into a self-taught venture in python... If I can't get this right the first time, it worries me a little.

MISSING=object()

def __getitem__(self, key=MISSING, time=MISSING, money=MISSING): if time is MISSING or money is MISSING: if time is not MISSING and key is not MISSING: money = key elif money is not MISSING and key is not MISSING: time = key else: time, money = key

Is this right? Wrong? The hard way? The slow way?

What about when there are three arguments? What then?

I'm using () rather than a MISSING as the sentinel here because it makes the code simpler. For the same reason, if we need to choose a key to pass in by default for a[**kwargs only] without defining a () in __getitem__, I have consistently advocated using () for this. Likewise, *not* having a default [or requiring the method to define its own default] key presents a slight problem for __setitem__, illustrated below. def __getitem__(self, key=(), /, **kwargs): return self._getitem(*(key if isinstance(key, tuple) else (key,)), **kwargs) def __setitem__(self, key=(), value=???, /, **kwargs): return self._setitem(value, *(key if isinstance(key, tuple) else (key,)), **kwargs) def _getitem(self, time, money): ... def _setitem(self, value, time, money): ... [delitem the same as getitem] Basically, you can easily write code that leverages the existing function argument parsing, simply by performing a function call.

On Wed, 2 Sep 2020 at 05:20, Christopher Barker wrote:

And, in fact, the current sytax is pretty tricky as well. Say you want to make a class that takes multiple indices -- like a Mtraix, for instance.

your __getitem__ looks like:

def __getitem__(self, index):

as they all do. But what might index be?

It could be one of: * an object with an __index__ method -- so a single index, simple * a slice object * a tuple, where each item in the tuple could be both of the above.

And this is the reason why personally I would prefer to add a new dunder with more defined semantics. This poor sod of a method has already a lot to handle, exactly because it's behaving nothing like a function. As a programmer developing that method, you are essentially doing the job of the interpreter.

But that's all OK, as the number of people that need to write these methods is small, and the number of people that can use the feature is large.

True but there's a point where one should question if the direction getitem has taken is the right one. -- Kind regards, Stefano Borini

Chris Angelico writes:

Possibly a better example would be path division in Python,

I would have chosen '+' for that operation for the same reason '+' "just works" for sequences. It's a very specialized use case but 5 * pathlib.Path('..') / "foo" makes sense as an operation but oh, the cognitive dissonance! It's so specialized that <0.99 wink>. '/' is fine for pathlib, and I trust Antoine's intuition that it will "work" for (rather than against ;-) pathlib users.

or stream left/right shift in C++.

I'm very sympathetic to your argument for this one.

Is that abuse of notation or something else?

As I wrote earlier, I don't think "abuse of notation" is a useful analogy here.

Whatever it is, it's not "operator overloading" in its normal sense; division is normally the inverse of multiplication, but there's no way you can multiply that by "quux" to undo that last operation,

But if you spell it '+', pathlib.Path("/") + "foo" + "bar" + "quux" - "bar" makes some sense, modulo the definition of the case with multiple "bar". I don't argue it's useful, just interpretable without head explosion.

Maybe we need a different term for this kind of overloading, where we're not even TRYING to follow the normal semantics for that operation, but are just doing something because it "feels right".

"Overloading" for this case doesn't bother me, but if you want to introduce a new term, Greg Ewing's "operator repurposing" WFM. Steve