What about having a .get(index, default) method for arrays like we have for dicts?
When I need to traverse nested dicts, is a common pattern to do somedict.get('foo', {}).get('bar', {}) But there is no such equivalent for arrays, wouldn't be nice if we can follow somedict.get('foo', {}).get('bar', []).get(10) ... ? What I do in this case is surround with try/except IndexError and set some variable to None, or wrap the try/except block in a function. But this is noise in my opinion, I just want to follow the same reasoning that I have with dicts: I'm searching for some path deep nested, and want a default if not found. What do you think? Regards -- “If you're going to try, go all the way. Otherwise, don't even start. ..." Charles Bukowski
I think I'm missing something. Daniel wants a `list.get` method with similar semantics to `dict.get`. So instead of writing: ``` try: x = lst[i] except IndexError: x = default ``` one could write `x = lst.get(i, default)`. How would you rewrite that with PEP 505? I've also wanted this a couple of times, although I can't remember the reason. It's not just about traversing incomplete nested data structures. I don't see much disadvantage since the analogy with `dict.get` would make it very easy to learn and understand, to the point that I've wondered why this doesn't exist already. On Sat, Jun 27, 2020 at 5:09 PM Guido van Rossum <guido@python.org> wrote:
Em ter., 30 de jun. de 2020 às 15:49, Alex Hall <alex.mojaki@gmail.com> escreveu:
With PEP 505 is not possible to have an arbitrary default as the result of null coalesced expression will be None if a None is found. The lst.get(i) would be rewritten as lst?[i], and composing dct.get('foo', []).get(0, []).get('bar') would be something like dct?['foo']?[0]?['bar'], from what I read, this is very terse and terse is good in most cases good in my opinion. It removes the error handling noise by replacing it by one char I'm repeating myself here, but again, the use case is to fetch deedly nested data. All this talk make me notice that IndexError, and KeyError are LookupError instances but AttributeError is not, still pep 505 have an elegant solution for all these three. Also it handles the None?.method() case, when something return an object or None and you want to call a method on this if is not None. I think we should put more effort on pep 505
-- “If you're going to try, go all the way. Otherwise, don't even start. ..." Charles Bukowski
On Thu, Jul 2, 2020 at 9:33 PM Daniel. <danielhilst@gmail.com> wrote:
No, that's not right. `dct?['foo']` translates to `dct['foo'] if dct is not None else None`, which is very different from `dct.get('foo')`. Similarly `lst?[i]` is for when `lst` might be None, not empty. All PEP 505 gives you in the case where keys might be missing or lists might be empty is that you can write `.get` without defaults, e.g. `dct.get('foo')?.get(0)?.get('bar')`. I would quite like to have PEP 505, but I don't know how to revive it. And I'm not surprised that it's deferred, there's generally reluctance to add new syntax, especially symbols. On the other hand, `list.get` seems very doable to me. It's not new syntax. It would be extremely easy to learn for anyone familiar with `dict.get`, which is pretty much essential knowledge. You'd probably have some people guessing it exists and using it correctly without even seeing it first in other code or documentation. I haven't seen anyone in this thread suggesting any cost or downside of adding the method, just people asking if it would be useful. I feel like I answered that question pretty thoroughly, then the thread went quiet.
Alex Hall writes:
Bottom line: new use cases or modified semantics that are more attractive to people who don't sympathize with the "in NOSQL objects often aren't there" use case. Some handwavy discussion below. Note that Nick Coghlan in the PEP 622 thread over on python-dev is proposing a very different use for '?' which seems likely to conflict. It wouldn't rule out syntax, but definitely would make it more difficult to use '?' (though probably not syntactically impossible) for this purpose.
Unfortunately, the people whose answers you really want to hear don't hang out here much any more. Steven d'A has remarkably comprehensive knowledge, but he's not as good at channeling Guido on what isn't yet as he is at explaining what is already. Not sure where Andrew Barnert has got to recently, but put him and Steven together and you get the best of both that's available here. Greg Ewing is another valuable source, but his design instincts often are quite different from Guido's. Lots of good ideas and discussion from these developers, but add all that up and you don't get Guido or Barry or Nick. Not necessarily *worse*, but different, and if you can't show that some feature requires a horrible hack or inordinate amounts of hard-to-get-right trickery, being different from Guido still connotes being un-Pythonic, and that's hard to get around. (Don't let that stop you from arguing with Guido -- among his other remarkable characteristics, he's been known to change his mind and accept features he long opposed. :-) I don't suggest that you try to open up a thread on python-dev where the folks who dropped the acid in the Kool-Aid do show up frequently. All of the reasons you and others give for wanting this are well- known, and they just didn't do it. Until you've got a powerful new argument or different, more attractive semantics, you'll just get shooed back here. I can say that it's not just the high cost of a symbol that tabled that PEP. Chained accesses like "dct['foo'][0]['bar'] are just expressions, but among the old-timers there's a deep suspicion of the "fluent" style of programming, chaining method calls by returning self. This isn't the same thing, but "null coalescing" operators (I prefer "null propagating" but "coalescing" seems to be the preferred term among proponents) has something of the same flavor, albeit only for None. It necessarily privileges None[1], guaranteeing ambiguity of that value, especially in a context like parsing JSON, where JSON null is typically translated to Python None. There was also a concern that null coalescing encourages hard to debug, unmaintainable programming. I don't think anybody provided hard evidence on that count, but introducing an operator specifically to *postpone* detection of exceptional conditions just doesn't sit well with a lot of people. It confuses things that *should* be there but aren't, with things that *might* be there but aren't, and with None when it's actually there. Postponing error detection can be done with 'try', but try doesn't confuse those things.[2] The principle of "consenting adults" means that possibility of misuse won't kill an otherwise good idea, but the question "isn't there a more Pythonic, explicit semantics that doesn't allow errors to propagate?" is one way to get an idea tabled. Finally, once you compare null-coalescing operators with 'try', it's clear that 'try' is far more powerful as well as sufficient for this application, and it's just not clear that null-coalescing is sufficiently special a special case to deserve syntax. Don't @ me. That is, I don't have a horse in this race (except that it would be syntax I personally am unlikely to ever use, and probably won't even see very often). I'm just trying to summarize some of the themes of the discussion FYI. Footnotes: [1] Addition of a special NULL object was proposed, IIRC, but I forget why that was considered a bad idea. [2] On the other hand, ':=' got in even though that is what '=' is for. That's a precedent that PEP 505 didn't have, turning a statement into an expression.
On Fri, Jul 3, 2020 at 5:36 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Thanks Stephen, it's nice to get some idea of what happened to PEP 505. It would be great if someone could put an official summary in the PEP, like a "Deferral notice". However it feels like you misread my email. You quote me saying this:
but then apparently go on to talk about PEP 505. At that point I was talking about the low cost of a `list.get` method, in contrast to PEP 505.
On Thu, Jul 2, 2020 at 10:33 PM Alex Hall <alex.mojaki@gmail.com> wrote:
I just had a coworker ask if there was something akin to `list.get(0)`, so I'd like to try to revive this. I propose that: 1. There is basically no cost in terms of mental capacity, learnability, or API bloat in adding list.get because of the similarity to dict.get. 2. There are plenty of times it would be useful, as seen in https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC... 3. If the above two points are true, we should go ahead and add this. I think that the discussion here simply fizzled away because: 1. People got distracted by talking about PEP 505 which really isn't very relevant and would solve a different problem. 2. There are no major objections, so there isn't much left to talk about, which seems like a silly way for a proposal to die. The only decent objection I saw was skepticism about valid and frequent use cases but once I answered that no one pursued the matter.
I just came across this again while implementing an parser I would like to compare stack elements as if stack[-3] == x and stack[-2] == y and stack[-1] == z and somewere below elif stack[-1] == t I had to spread `len(stack)` in a lot of places. People said about the length of a list is usually known, but when you use it as a stack is the oposit. Em ter, 25 de ago de 2020 09:44, Alex Hall <alex.mojaki@gmail.com> escreveu:
Here is one example added in stdlib https://github.com/python/cpython/pull/14813/files It's checking the truthness of value before calling value[0], this is a perfect match for value.get(0) Here is another, just the same use case, https://github.com/python/cpython/pull/17515/files When you deal with parsing problems yielding sequence of tokens are pretty common and knowing how much tokens are there is a matter of input. When you deal with recursives problems using a stack too.. so there are use cases out there and it's a tiny change Em ter, 25 de ago de 2020 09:52, Daniel. <danielhilst@gmail.com> escreveu:
I mean you could write these as: if stack[-3:] == [x, y, z] and elif stack[-1:] == [t] But plenty of use cases still exist ( https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC...) and I think we shouldn't need to keep talking about them. On Tue, Aug 25, 2020 at 2:54 PM Daniel. <danielhilst@gmail.com> wrote:
On 2020-08-25 13:52, Daniel. wrote:
if stack[-3 : ] == [x, y, z]:
and somewere below
elif stack[-1] == t
elif stack[-1 : ] == [t]:
I had to spread `len(stack)` in a lot of places.
You can use slicing to reduce the number of `len(stack)`.
People said about the length of a list is usually known, but when you use it as a stack is the oposit.
[snip]
On Tue, Aug 25, 2020 at 5:50 AM Alex Hall <alex.mojaki@gmail.com> wrote:
A lot of ideas fizzle away on this list. The fact is that it's a lot of work to get a new feature accepted into Python, even if it's not very controversial -- it's the classing "scratching an itch" problem -- someone needs to find it itchy enough to do a LOT of scratching -- any idea needs a champion that will carry it though. This one is easier than most because it's pretty much a do we or don't we question, with the spelling semantics, all pretty much already decided. Though I haven't quite seen it said explicitly -- is this proposed to be added to the Sequence ABC? If so, there is a potential problem -- the ABC's "namespace" is not reserved, so if you add .get(), then any code out in the while that is already using .get in a custom sequence could potentially break. See the discussion about the idea of adding a .view() method to Sequences -- same issue. Even of you think it's a good idea, it's still hard to add a new (non-dunder) method to an ABC. -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Tue, Aug 25, 2020 at 7:23 PM Christopher Barker <pythonchb@gmail.com> wrote:
I think if it's a potential problem to add it to the ABC then let's just defer that and only add it to the built in sequences. But I'm not clear on what the problem is. If you have some code like this: ``` class MySequence(Sequence): def get(self, i): ... MySequence().get(3) ``` and then add .get to the Sequence ABC, the existing code will not be immediately broken because the custom MySequence.get overrides the ABC method so everything behaves as before. A problem that I can see eventually arising is if someone writes some generic code for Sequences using .get, MySequence won't work with it. That won't happen immediately and I'm not sure if I'd call it a break in compatibility. But I can see that it is a problem. Maybe if we build up enough methods to add to the collection ABCs (Sequence.view, Sequence.get, Set.intersection/union/etc, Mapping.__[i]or__) we can justify adding them all at once in Python 4 or in a new module collections.abc.v2 which has subclasses of the originals.
On Tue, Aug 25, 2020 at 10:51 AM Alex Hall <alex.mojaki@gmail.com> wrote:
On Tue, Aug 25, 2020 at 7:23 PM Christopher Barker
I think if it's a potential problem to add it to the ABC then let's just
defer that and only add it to the built in sequences.
Well yes, that’s a fine solution. Though an advantage of adding it into the ABC is that it could be a mixin, and all Sequences would them get it. But I'm not clear on what the problem is. If you have some code like this:
Yes, but then the third party object that was a Sequence may no longer be one. Maybe if we build up enough methods to add to the collection ABCs
(Sequence.view, Sequence.get, Set.intersection/union/etc, Mapping.__[i]or__) we can justify adding them all at once in Python 4
Careful now! No one wants to repeat the py2/3 transition! or in a new module collections.abc.v2 which has subclasses of the originals.
Maybe, but the trick is: how do you “build up” thus stuff before you have a place to put it? And what’s the point of a place to put new stuff, when there isn’t any new stuff? -CHB
-- Christopher Barker, PhD
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
As a point of interest, is get() considered an official part of the mapping protocol, or just nice-to-have? The docs don't seem to be very clear about that. There used to be some tables listing the methods making up the core sequence and mapping protocols, but I can't find them now. Have they been removed? Are the ABCs now the definitions of the protocols? -- Greg
On Wed, Aug 26, 2020 at 1:47 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I would also like to see an official answer to this. Are the mixin methods of the ABCs considered part of their contract? Is it documented somewhere? I think it should be. On the one hand it sort of feels like the contract is just the abstract methods that users are supposed to implement themselves, and the mixin methods are just a convenient benefit of subclassing. It's hard to say where that feeling comes from, but I think the name "mixin method" is part of it. I'm glad to see I'm not the only one that gets this feeling. On the other hand, if I have a variable of type Mapping, I'm usually going to assume it has a .get method that behaves like dict. Most tools will assume that too. If it doesn't, I'll be quite surprised and probably annoyed.
The mixins are part of the contract. You are free to override them, their implementation is just a helpful,default. FWIW I don’t think we should add a .get() method for Sequence or for list. It reeks of hyper-correctness. On Wed, Aug 26, 2020 at 01:33 Alex Hall <alex.mojaki@gmail.com> wrote:
On 27/08/20 2:32 am, Guido van Rossum wrote:
The mixins are part of the contract. You are free to override them, their implementation is just a helpful,default.
That doesn't really answer my question. To put it another way: If someone wanted to implement an object that obeys the mapping protocol, but without inheriting from the Mapping ABC, would they be obliged to provide a get() method? -- Greg
Yes. That's what I meant by "the mixins are part of the contract", sorry. On Wed, Aug 26, 2020 at 8:45 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Aug 26, 2020 at 4:32 PM Guido van Rossum <guido@python.org> wrote:
FWIW I don’t think we should add a .get() method for Sequence or for list. It reeks of hyper-correctness.
What do you mean by this? I want the method because it's useful and convenient, not because it's 'correct'. I don't think that lists are 'wrong' right now.
But for your convenience you are proposing a problematic change (see posts by others). Maybe I meant overgeneralization instead. On Wed, Aug 26, 2020 at 10:20 AM Alex Hall <alex.mojaki@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Aug 26, 2020 at 7:30 PM Guido van Rossum <guido@python.org> wrote:
But for your convenience you are proposing a problematic change (see posts by others).
I think we've established it's problematic to add it to the Sequence ABC, but in terms of only adding it to specific sequences like lists, I don't know what other posts you're referring to. Why is it problematic?
Maybe I meant overgeneralization instead.
Again, the goal isn't to generalize. I'm not trying to make lists look like dicts. It's not even possible to get close - `if i in x: print(x[i])` does very different things for lists and dicts.
On Wed, Aug 26, 2020 at 10:45 AM Alex Hall <alex.mojaki@gmail.com> wrote:
It would break duck typing -- if you check for IndexError you can accept pretty much any sequence. But if you use .get() you can only work with lists. This is why we have ABCs in the first place (and static checking). So that you can't add it to Sequence is already a smell. If it's in the Mapping protocol, why isn't it in the Sequence protocol? There's also the slippery-slope argument brought up earlier -- should it be added to tuple? To range? What should it do for a negative index? I also think it negatively affects readability. Today when I see `foo.get(bar)` I assume that (unless the code is full of tricks) foo is a dict or Mapping, and bar a key. But if list also grows a .get() method I have to assume in my head that foo is a list and then bar an int, and I basically have to read the rest of the code twice, once assuming foo is a Mapping, and once assuming it's a list.
The generalization I am talking about here is the assumption that because list and dict both support x[y], and dict also has x.get(y) which returns a default instead of raising, then it would be a good idea to add the latter also to list. And I judge it an overgeneralization because there's more to it than meets the eye. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Aug 26, 2020 at 7:56 PM Guido van Rossum <guido@python.org> wrote:
This is all just a general feature of ABCs. They were written once and frozen in place and no new methods can be added to them because it would break compatibility. It's not specific to this proposal. | and |= operators were added to dict but not to Mapping/MutableMapping for the same reason, even though the same operators exist with a similar meaning in the Set/MutableSet protocols. So there's a clear precedent for this. It's a little unfortunate that we can't put this in the Sequence ABC, but it's not a big deal. The vast majority of the time I'm interfacing with my own code and so I know that I'm working with a list, not some mysterious Sequence. I don't think people are going to change their public facing function signatures from `def foo(x: Sequence)` to `(x: list)` just so that they can use `x.get`. Even if they do, it suggests that they think `.get` is really useful, and callers can just convert their argument to a list first. Similarly adding the union operator to dict is fine because people usually use dicts, not generic Mappings. Besides, list already defines a bunch of stuff not found in Sequence or MutableSequence: copy, sort, `<`, `+`, and `*`. What's one more?
There's also the slippery-slope argument brought up earlier -- should it be added to tuple? To range?
I think Steven D'Aprano asked this. I don't see it as an argument against or a concern about a slippery slope, just a question. I think it'd be nice to have it on lots of sequences, but it's OK if it's not. Neither scenario strikes me as problematic. The union operator was also added to defaultdict, OrderedDict, and ChainMap. Was this a problem?
What should it do for a negative index?
I think most people expect that if `lst[i]` returns a value then `lst.get(i)` should return the same value, therefore it should return a default only when `lst[i]` fails. Sometimes you specifically want `lst.get(-1)` - there's a couple examples of that in [my earlier post]( https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC...) and Daniel recently presented a case when he wanted that. This isn't the ideal behaviour for all use cases, but only as much as normal indexing isn't the ideal behaviour for all use cases. Sometimes I've had bugs because I forgot to account for negative indices, but I'm still very grateful to you that negative indices work. We can't make something equally convenient for all use cases, but that doesn't diminish the value of this method. You said yourself that the generic alternative for all sequences is to "check for IndexError", a sign that doing the equivalent of that is more intuitive.
That's already a poor assumption to make. `get` is an extremely generic method name that has lots of meanings in different classes.
The purpose of `foo.get(bar)` is to get a value, not to tell you that foo is a dict. If the rest of the code isn't able to tell you whether `foo` is a dict or a list, then either the code is poorly written (and .get is not to blame) or it doesn't matter because you're just getting individual values. `foo[bar]` doesn't differentiate between lists and dicts and this is generally considered a good thing, not a problem.
I'm not assuming that list should have .get because dict has it. I'm claiming list should have it because it would often be useful, based on [empirical evidence]( https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC... ). Anyway, thank you for this. I'm enjoying the discussion and I'm glad to hear actual reasons that can be argued about rather than the proposal just fizzling away.
On Wed, Aug 26, 2020 at 12:38 PM Alex Hall <alex.mojaki@gmail.com> wrote:
and the Set ABC doesn't have the set methods, like .union, .intersection ... (though I still don't get why that is ...)
This is something to be thinking about, however -- having the builtins extend the ABCs hasn't been a big deal, what with dynamic typing and all -- but as folks start using type hints, and static type checking more, it could be an issue. Honestly, I haven't delved into type checking much, so maybe I'm making this up, but what if you write a function that is type hinted to take a Mapping, and uses | or |=, and all the test code uses a built in dict, and all seems to be well. Then this code starts being more widely used, and someone runs it with a different Mapping object, and it passes the static type checking pass, and then, oops, a run-time failure. It makes me wonder of the utility of the ABCs is the builtins are extending them. Maybe we need a set of (at least for testing) builtins that are fully ABC conformant ? -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Aug 26, 2020 at 10:10 PM Christopher Barker <pythonchb@gmail.com> wrote:
This made me curious so I did an experiment, but with Set because I couldn't be bothered to worry about Python versions. This code: ``` import typing import collections import builtins def foo(t: typing.Set, c: collections.Set, b: builtins.set): print(t + t) print(t | t) print(t.union(t)) print(c + c) print(c | c) print(c.union(c)) print(b + b) print(b | b) print(b.union(b)) ``` complains about `+` (as it should) and `c.union` in both PyCharm and mypy, and nothing else. So typing.Set is based on the builtin and not the ABC. If you want to support any subclass of collections.Set, your type hint needs to be collections.Set. Using typing.Set will fail to show warnings when you use a builtin only method, and will show warnings when you try to assign an ABC set (e.g. I get a warning with `foo(t=<instance of collections.abc.Set>)`. The problem then is that `collections.abc.Set[int]` doesn't work, i.e. you need to use typing.Set if you want generics.
Use typing.AbstractSet for the ABC (there was an unfortunate naming conflict -- we have dict and Mapping, list and Sequence, but set and Set...) On Wed, Aug 26, 2020 at 2:16 PM Alex Hall <alex.mojaki@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Oh thanks, I had no idea that existed. I wanted to experiment with Mapping and unions but it's not working. I couldn't find any existing issues about this. Should I be opening an issue? If not then there's no need to explain what I'm doing wrong. ``` $ cat test.py print({} | {}) $ ./python test.py {} $ mypy --python-version 3.9 test.py test.py:1: error: Unsupported left operand type for | ("Dict[<nothing>, <nothing>]") Found 1 error in 1 file (checked 1 source file) $ mypy --version mypy 0.782 ``` On Wed, Aug 26, 2020 at 11:18 PM Guido van Rossum <guido@python.org> wrote:
I think this method would be useful for list itself (maybe tuple, but I care less about that). It certainly should not be part of the Sequence protocol. I have often programmed a pattern where I .append() elements to a list in a loop, and *expect* at least N things to make it in. In such case, getting a sentinel answer to mylist.get(N-1) would be convenient. Obviously, I've lived without that fine. But it's a little bit shorter and more expressive than: x = mylist[N-1] if len(mylist) >= N else None On Wed, Aug 26, 2020, 1:30 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Aug 25, 2020 at 10:22:20AM -0700, Christopher Barker wrote:
This one is easier than most because it's pretty much a do we or don't we question, with the spelling semantics, all pretty much already decided.
Is it to be added to lists, or lists and tuples? How about range objects? range(1, 9, 2).get(100) Strings and bytes? "abc".get(100) Do we require 3rd party sequences to add this as well, in order to continue being called sequences?
Though I haven't quite seen it said explicitly -- is this proposed to be added to the Sequence ABC?
If so, do we require that it return None as the default, or are types permitted to return whatever missing value they see fit? -- Steve
On Wed, Aug 26, 2020 at 1:41 AM Steven D'Aprano <steve@pearwood.info> wrote:
I think at a minimum it should be in lists and tuples. I can't see any reason not to add it to range, strings, bytes, deque, array, etc. beyond implementation effort. I also think it's fine to leave them out at first.
Do we require 3rd party sequences to add this as well, in order to continue being called sequences?
I think this is pretty much the same question as the discussion about ABCs. It seems like the answer is "no", or at least "not yet".
This should be identical to Mapping.get. None isn't required to be the default missing value - callers can specify any value they want - but None is the default default, i.e. `def get(self, item, default=None)`. I don't know of any Mapping or mapping-like class that has a different default default. If such classes exist, then I guess None is just the default default default :)
Alex Hall writes:
If the "keyword arguments in __getitem__" feature is added, .get() is purely redundant. (Of course this thread would then become "make 'default' a standard keyword argument for mutable collections.") Since .get() doesn't seem to be terribly useful, and there is quite a bit of support for keyword arguments in __getitem__, I think this can be tabled pending resolution of that proposal. Steve
On Wed, Aug 26, 2020 at 5:00 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Is that something people want to do? Do people want to be able to write `my_dict[key, default=0]` instead of `my_dict.get(key, 0)`? What about `my_dict[key, default=None]` instead of `my_dict.get(key)`?
Alex Hall writes:
Of course they don't want to do that. Of course dict.get is going nowhere. Of course that's redundant. Of course if collections get a standard 'default' for __getitem__, many people *will* start writing `my_dict[key, default=None]`, if only because they don't read enough docs to know about dict.get.
On Thu, Aug 27, 2020 at 10:15 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
OK, I'll try again. Do people want collections to get a standard 'default' for `__getitem__`? I don't want mappings to grow a second way to do the same thing, and I don't want sequences to have a different way to do it from mappings.
An .get method wont hurt duck typing, it makes it better. Also most of the time we're working with plain dicts and lists not abtract classes. If you're typechecking for a Sequence you cant treat it as a list, and this is already true because there are methods in list that dont exists Sequence as Alex said. Also the idea was always for a list, tuples and ranges are okay, but tuples are immutable and ranges are constructed so it's not that useful, as it is for lists derived from some unknown input It's simple: def get(self, i, default=None): try: return self[i] except IndexError: return default That was the idea, just a safe wrapper arround IndexError. Because it's very common to need to catch this. I posted two bugs that happened because of uncaught IndexErrors. .get() enforces better coding, it's more readable, avoid distracting indentation blocks and flow jumping, the error handling is reduced and more explicity and more fluid, you can get a fine default if you want. Besides breaking Sequences contract on list, (which is already broken as Alex said, by .copy, < and other methods) I cant see any other problem for adding it to *lists* Em qui, 27 de ago de 2020 05:57, Alex Hall <alex.mojaki@gmail.com> escreveu:
On 27.08.2020 14:40, Daniel. wrote:
I disagree on the above assessment. I have had such a get() builtin in mxTools for more than 20 years now and found that I hardly ever used it: https://www.egenix.com/products/python/mxBase/mxTools/doc/#_Toc293606201 The reason is simple: unlike for dicts, where you often expect non-existing items (e.g. in keyword arguments, config files, etc.), having a missing index position in a list which you parse is almost always an error. Now, errors should be clearly marked and handled as such, hence having a try-except is the better coding strategy. For those cases, where a list can have a variable number of entries (e.g. optional arguments, file lists, etc.), code should clearly branch on list length and then determine the right strategy to fetch items. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Aug 27, 2020 at 3:09 PM M.-A. Lemburg <mal@egenix.com> wrote:
I'm copying my earlier post because apparently linking to it repeatedly has no effect and people keep on claiming that this method wouldn't be useful while not saying a word about the evidence I presented that it would be. dict.get is definitely much more useful. But list.get is still useful reasonably often. You can see quite a bit of demand for the method here: https://stackoverflow.com/questions/5125619/why-doesnt-list-have-safe-get-me... Below are some use cases I found and how they could be refactored with .get. Apart from the first one, they were all found simply by grepping for `except IndexError`. It's pretty easy to pick obvious use cases out from there. In some cases readability suffers a bit and using .get might not be the best idea, but I've included them anyway so that we can see what over-eager usage of the method might look like. ``` _pytest/_code/source.py if insert_index >= len(values): end = None else: end = values[insert_index] end = values.get(insert_index) -------------------- prompt_toolkit/layout/layout.py try: return self._stack[-2].content except IndexError: return self._stack[-1].content return self._stack.get(-2, self._stack[-1]).content -------------------- prompt_toolkit/buffer.py try: word = words[state.n] except IndexError: word = "" word = words.get(state.n, "") -------------------- prompt_toolkit/widgets/menus.py try: selected_item = self.selected_menu[level + 1] except IndexError: selected_item = -1 selected_item = self.selected_menu.get(level + 1, -1) -------------------- prompt_toolkit/document.py try: return self.text[self.cursor_position + offset] except IndexError: return "" return self.text.get(self.cursor_position + offset, "") -------------------- prompt_toolkit/contrib/regular_languages/lexer.py try: return lines[lineno] except IndexError: return [] return lines.get(lineno, []) -------------------- pip/_internal/cli/autocompletion.py try: subcommand_name = [w for w in cwords if w in subcommands][0] except IndexError: subcommand_name = None subcommand_name = [w for w in cwords if w in subcommands].get(0) -------------------- pip/_internal/cli/autocompletion.py try: current = cwords[cword - 1] except IndexError: current = '' current = cwords.get(cword - 1, '') -------------------- pip/_internal/configuration.py try: return self._get_parser_to_modify()[0] except IndexError: return None return self._get_parser_to_modify().get(0) -------------------- pip/_vendor/distlib/resources.py try: result = self.index[i].startswith(path) except IndexError: result = False result = self.index.get(i, "").startswith(path) -------------------- stack_data/core.py try: return not self.lines[i - 1].strip() except IndexError: return False return not self.lines.get(i - 1, "not blank").strip() -------------------- parso/tree.py try: return self.parent.children[i + 1] except IndexError: return None return self.parent.children.get(i + 1) -------------------- ipython/IPython/core/completer.py try: return self.matches[state] except IndexError: return None return self.matches.get(state) -------------------- ipython/IPython/core/magics/basic.py mode = '' try: mode = parameter_s.split()[0][1:] except IndexError: pass mode = parameter_s.split().get(0, '')[1:] -------------------- ipython/IPython/core/tests/simpleerr.py try: mode = sys.argv[1] except IndexError: mode = 'div' mode = sys.argv.get(1, 'div') -------------------- ipython/IPython/utils/text.py try: tgt = parts[field] return tgt except IndexError: return "" return parts.get(field, "") -------------------- py/_xmlgen.py try: tag.parent = self.parents[-1] except IndexError: tag.parent = None tag.parent = self.parents.get(-1, None) -------------------- ipython/IPython/core/tests/tclass.py try: name = sys.argv[1] except IndexError: pass else: if name.startswith('C'): c = C(name) name = sys.argv.get(1, "") if name.startswith('C'): c = C(name) ```
On 27.08.2020 15:17, Alex Hall wrote:
If I'm not mistaken, all those listed cases fall under the second case I mentioned (branching on length). If your API allows for optional list entries, then it's better to branch on length (e.g. as in the pytest case). If this happens only in exceptional cases, use try-except (most other cases you listed). Both coding styles make the intent clear without doubt. As mentioned before, I had access to such a builtin for many years, but hardly ever chose to use it and instead opted for the more explicit branching on length. IMO, being able to write something with fewer key strokes is not always a good idea. Unless you write code only for yourself, it's usually better to emphasize on intent rather than brevity. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Aug 27, 2020 at 3:37 PM M.-A. Lemburg <mal@egenix.com> wrote:
All but the first example, i.e. all the uses of try-except, were found by grepping for `except IndexError`. That was just the easiest method I found to find examples. The fact that try-except was used in those cases does not necessarily mean that the index is missing "only in exceptional cases". Perhaps because you like to emphasise intent when you're coding you assume that "exceptional" is the intent when you see the keyword "except". It's not how I interpret these examples. I think truly exceptional cases would have a change in behaviour like showing a warning. All that's happening here is getting a default value, and using try-except is the easiest available syntax. Often the use case is that the list may be empty or you're at the edge of the list and you need to go one beyond. That's normal and expected, not exceptional. If list.get existed then probably it would have been used in many of these examples. But of course it didn't exist and I don't have a way of searching for custom get functions so nearly all the examples use except which you take to mean exceptional.
On 27.08.2020 16:01, Alex Hall wrote:
Could be that authors thought try-except to be the easier to use syntax, even when the length check is less effort to write, faster and clearer in stating that the missing items are expected. If those quoted code snippets really all refer to non-exceptional cases, it would be better to guide people to the length branching approach. And you could be right that some authors may have opted for list.get(), but the point of implementing such changes goes somewhat beyond "it reduced number of keystrokes". If it results in people writing less obvious code, it's usually not a good idea. Take e.g. the dict.get() interface. We should have not given it a default argument of default=None, since it's not obvious that mydict.get(x) will return None in case x is not part of mydict. Instead, it would have been better to require the default argument, so that you have to write mydict.get(x, None) or even mydict.get(x, default=None). That ship has sailed, but it should be warning to think twice about adding such apparently useful things. I had thought mx.Tools.get() would be useful at the time and it was easy to implement, along with several other additions. Many of them did not get much use. Others ended up in similar form in itertools or operator. The most useful addition was irange(), which I used a lot. Others had very similar needs and eventually enumerate() was added to the builtins. BTW: I also think that the analogy between dict.get() and list.get() is bit off... Asking a dict d of potentially any number of items for existence of a particular item x is somewhat different than asking a list l of a certain length i for the item at position i+1. The programmer should know that there is no item at position i+1. However, it's not obvious before trying that item x doesn't have a mapping in d by just looking at the object properties of d. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Aug 27, 2020 at 11:37 AM M.-A. Lemburg <mal@egenix.com> wrote:
Suppose we didn't have dict.get(). I would then probably write: val = mydict[key] if key in mydict else None Likewise, since we don't have list.get(), I would write: val = mylist[N] if len(mylist) >- N-1 else None Neither of those is impractical, but in both cases .get(key) seems to express the intent in a more obvious way. That said, it is trivial to write a get() function that does the same thing... and I've never bothered to do so. In fact, I could write a get() function that was polymorphic among mappings and sequences with very little work, which I've also never done. So I guess my case for the huge importance is undercut :-). -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On 27.08.2020 17:53, David Mertz wrote:
Really ? Let me try again :-) Getting an item from a bag of unknown items can easily fail, because you don't what's in the bag until you try to find the item. Trying to access an item from a non-existing bucket in an array will always fail. You know that in advance, since the bucket doesn't exist as per the array properties. You don't have to try finding it first. Not sure whether I'm getting the difference across. Grabbing into the void is different than grabbing into the dark :-) You might actually find something in the dark.
Well, I did a long time ago and hardly ever used it. For me, both are signs of "looks useful on the outside, but isn't in reality". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
M.-A. Lemburg writes:
On 27.08.2020 17:53, David Mertz wrote:
I think David is comparing the .get syntax to the if-else syntax. First of all, he got the if-else wrong, which is a sign that .get is easier to use, at least for writers and quite likely for readers. Second I agree that ".get" is easier to read and understand (especially with explicit default, as you advocate). It's even more stark if you're after a component of the list element: val = mylist[N if len(mylist) > N else default].content vs. val = mylist.get(N, default).content
But this isn't a C array. You don't know the length from the syntax. So you have to "do work" to discover the bucket isn't there before you access -- or you can delegate to the interpreter and catch IndexError. Given how precise IndexError is (it only catches integer arguments that fall outside the range of the array), I don't really see the argument for refusing to encapsulate the try in a .get method, or that its semantics differ from explicit bounds-checking. It seems to me that you're expressing a preference for "look before you leap" over "easier to ask forgiveness than permission" in the case of array indicies, which is a stylistic thing. It's a style I prefer in this case, but I can also see how other people would prefer a different style, one that makes sense to encapsulate in .get. Is there something more than stylistic preference that I'm missing?
On 28.08.2020 10:30, Stephen J. Turnbull wrote:
I was trying to explain why dict.get() is different than the suggested list.get(). dict.get() was added since the lookup is expensive and you want to avoid having to do this twice in the common case where the element does exist. It was not added as a way to hide away an exception, but instead to bypass having to generate this exception in the first place. dict.setdefault() has a similar motivation. list.get() merely safes you a line of code (or perhaps a few more depending on how you format things), hiding away an exception in case the requested index does not exist. If that's all you want, you're better off writing a helper which hides the exception for you. I argue that making it explicit that you're expecting two (or more) different list lengths in your code results in more intuitive and maintainable code, rather than catching IndexErrors (regardless of whether you hide them in a method, a helper, or handle them directly). So this is more than just style, it's about clarity of intent. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 28 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Fri, Aug 28, 2020 at 4:30 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
It was an unintentional mistake, but Stephen is right it makes the point. I fixed above, both my thinko reversing the N vs. N-1, and also the simple typo of `>-` instead of `>=`. No language change is going to fix the fact the '-' and '=' keys are next to each other on my keyboard. I think the comparison (with right syntax and semantics) is fair. Whatever the ontological significance MAL notes about "reaching into the void" versus "reaching into the dark", the code I would actually write if dict.get() didn't exist is pretty much the above. And it looks a whole lot like the code I actually do write for possibly-missing index positions. As a side note, I don't really get why everyone else thinks a try/except is the most natural approach while a ternary seems more obvious to me for this situation. But it kinda connects to me liking list.get() better, I think... since "not enough items" doesn't seem as *exceptional* to me as it apparently does to some others. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On 8/28/20 8:24 AM, David Mertz wrote:
try/except is specifically looking for the error in question, index out of bounds. The ternary requires restating in other words the bounds limit. For instance, for a list of N items, the allowable subscripts are -N to N-1, and the sample ternary is only checking for the upper bound, not the lower, so fails an too small of an index (maybe you know you are only using positive indexes?) Missing this won't happen with try/except. -- Richard Damon
On 2020-08-28 at 08:41:04 -0400, Regarding "[Python-ideas] Re: What about having a .get(index, default) method for arrays like we have for dicts?," Richard Damon <Richard@Damon-Family.org> wrote:
Without more context, I've lost the forest for the trees. There are multiple ways to do this, and the "best" way likely depends on the context and not the readability, writability, maintainability, or expressivity of that exact line of code. Where did the index come from? How/when/where was the list constructed? Could this "problem" have been recognized and/or addressed sooner? What, exactly, are the conditions that cause the index being out of bounds at this point in the logic, and what are the semantics now that it's happened? Given the preceding logic, is an index that's too small (i.e. mathematically less than the opposite of the lenght of the list) even possible? I didn't find either of the examples (one using an if/else ternary and one using a hypothetical list.get method) particulary easy to read or particularly indicative of an obvious use case. And at that point, re-writing one [presumably correct] line of code into another won't make or break anything.
On Fri, 28 Aug 2020 at 13:26, David Mertz <mertz@gnosis.cx> wrote:
As a side note, I don't really get why everyone else thinks a try/except is the most natural approach while a ternary seems more obvious to me for this situation. But it kinda connects to me liking list.get() better, I think... since "not enough items" doesn't seem as *exceptional* to me as it apparently does to some others.
Possibly because in reality, people very rarely write code like that, so they are working out what they'd use "from scratch", rather than just quoting a well-known idiom that they are familiar with? Which to me says that this method isn't nearly as useful in practice as some people are claiming. Paul
Em qui, 27 de ago de 2020 10:37, M.-A. Lemburg <mal@egenix.com> escreveu:
In this case is pretty obvious the intent and this is an know abstraction. Avoid brevity will not make it cleaner. IMHO, explicitng index bounds and try/catch are distracting noise which hides intent instead. Also .get() would not prevent anybody from explicity checking bounds or catching index exceptions when appropriate
On Thu, Aug 27, 2020 at 03:17:24PM +0200, Alex Hall wrote:
Perhaps it's just that people don't think that the examples are compelling. Or they are burning out and finding it hard to care. Personally, I found your examples underwhelming because they're mostly repetitions of the same pattern. I'd find it more interesting if there were a larger variety of cases, not merely a larger number, and I'd find it *really* interesting if you found a couple of examples of bugs caused by the common idioms. E.g. the slicing and "branch of length" idioms are tricky to get right if the index is negative. Find a couple of bugs in publicly visible code because people forgot to take negative indices into account, and you might find that more people will sit up and take notice. -- Steve
On Thu, Aug 27, 2020 at 6:32 PM Steven D'Aprano <steve@pearwood.info> wrote:
Personally, I found your examples underwhelming because they're mostly repetitions of the same pattern.
That's surprising to me. When I see the same pattern over and over that's when I most want to refactor into a common function/method. The repetition is annoying and the solution is easy and obvious. If the cases were all different it'd be much more complicated and chances are there wouldn't be a simple function that covered all cases. Do you not usually refactor duplicated code? What *do* you refactor? I'd find it more interesting if there
were a larger variety of cases
This sort of sounds like the problem. The examples aren't meant to be interesting.
On Thu, Aug 27, 2020 at 07:10:38PM +0200, Alex Hall wrote:
The repetition doesn't go away just because you give it a name. It's just easier to manage.
Depends on what the code is. If it's as trivial as a try...except with a single operation inside each block, it's hardly worth the bother to add an extra level of indirection, an extra function, just to reduce what is effectively a two-liner (albeit one which is commonly spread over four lines) to a one-liner: # Before try: x = L[index] except IndexError: x = "" # After x = get(L, index, "") Save a line of code, replace it with one more function to read, document and test. Instead of having to read a two (four) line try...except block where everything is explicitly in front of my eyes, I have to read a fairly bland and generic-looking "get" function where I'm not sure *precisely* what it does witout reading the docs. Get what? Does it support negative indices? Can it fail? If I change L to a deque will it continue to work? I'm not *against* it. I think if lists already had the method I wouldn't object to its existence. I'm just finding it hard to care, and I think that's probably the main issue you are up against. This doesn't have a compelling benefit, it's quite marginal, at least in the use-case above. That's my opinion, others may disagree.
You're asking why people aren't interested in solving this issue, you wonder why few people are terribly interested in it, and now you're saying that its not supposed to be interesting. I think that explains the relative lack of interest :-) That's just my two cents though, others may feel differently. -- Steve
Alex Hall writes:
Do you not usually refactor duplicated code?
Taking "you" as generic: it depends.
What *do* you refactor?
Code is expressive. In cases where the duplication results in equivalent objects, I habitually refactor. When the process is important to express and the duplication is syntactic, I will prefer an explicit but idiomatic syntax repeated in unrelated contexts. Thus I generally wouldn't consider your list to be "duplicated code that cries out for refactoring." When the process is complex then I'd refactor. Detecting an out of bounds index is complex enough because of fenceposts and negative indicies, although I probably wouldn't notice the negative indicies until they bit me (and in that case wouldn't refactor until bitten).
"You [both] keep using that word but I do not think [Steven] means what you think it means." (pace, Inigo) I find the long list of examples with mostly identical syntax *unpersuasive* for two reasons. First, if you're going to bounds check array indicies explicitly (rather than allow IndexError to "bubble up"), we already have a facility for doing that precisely and accurately: catch IndexError. That catches out of bounds indicies and nothing else[1], and it handles negative indicies correctly. list.get can't be implemented much more efficiently than by using try ... except IndexError, and invoking it imposes attribute lookup and function call overhead so I guess you net approximately nothing on time, and a little bit of space. If you care so much about either time or space, you'd implement inline, something like: value = lst[index] if - (N := len(lst)) <= index < N else default try ... except IndexError is TOOWTDI now, it's as readable as .get and more readable than the inline version, so introducing list.get would be redundant. Second, I can see why people would want it (see my reply to MAL), I just don't want it myself, and implementing your own *correctly* as a function is trivial: def get(container, subscript, default): try: return container[subscript] except (IndexError, KeyError): return default It follows that I think dict.get is a YAGNI, at least since the introduction of KeyError, and definitely now that we can write y = mydict[x] if x in mydict else default Footnotes: [1] It turns out that slice bounds can't be out of bounds, they simply get truncated, in the degenerate case to the empty slice.
Em qui, 27 de ago de 2020 10:09, M.-A. Lemburg <mal@egenix.com> escreveu:
Almost always an error is not always an error and errors are not always exceptions, a missing index maybe something expected.
Branching has the same effect of try/catch, it will break code flow and make it more complex, so less readable. .get() is an nice abstraction to get something or return a default, well understood, it cant be less readable. I I came across this many times, many enought to come here to ask, and the use cases we're already showed, so I don't think, is useless would be a good argument. I understand that people disagreed on what is readable, but not that it rarely needed
Alex Hall writes:
OK, I'll try again. Do people want collections to get a standard 'default' for `__getitem__`?
I don't know.
I don't want mappings to grow a second way to do the same thing,
They're going to get one, is my reading of the "named indicies" thread. That is, it looks to me very likely that index notation (`x[i]`) is going to support keyword arguments (`x[i, k=v]`). If so, that's going to be a natural way to support a `default` argument. I imagine some people will choose it because they like it, and if there are enough people who do there will be pressure for it to be TOOWTDI.
and I don't want sequences to have a different way to do it from mappings.
I think you have to choose a second way for mappings to get, or have sequences do it differently, or have redundancy in sequence operations. If the index operator supports keyword arguments (I think it likely; YMMV), you either have to give dict a keyword argument or you have to give sequences a .get method. The former redundancy implied for dict can be justified by backward compatibility with dict and consistency with sequences. The latter is just an arbitrary consistency, and I doubt it will happen. That's just my guess; I have no power to make it go either way.
On Thu, Aug 27, 2020 at 10:51 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I support named indices. But I strongly oppose using them in list, tuple, or dict themselves. So `mylist[99, default=4]` would still be a syntax error (or maybe a different exception). I only want named indices for data structures where a clear and compelling use is demonstrated (like xarray). But allowing them syntactically allows third parties to play around in their own classes to see if they are helpful. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, 27 Aug 2020 at 16:21, David Mertz <mertz@gnosis.cx> wrote:
Agreed - allowing keywords in subscripts is OK (I'm personally ambivalent about the value of it, but if people want it, sure). But only for 3rd party classes. No-one has come up with a good use case for allowing them in existing builtin classes, and I'm -1 on doing so without a strong use case. Furthermore, I'd be very, very strongly opposed to having x[i, default=d] be a way of supplying a default result if i isn't present in *any* subscriptable class, and particularly in dict as an equivalent for x.get(i, d). It's redundant, confusing, and unlike any other language I'm familiar with (all IMO, if that's not obvious...). Paul
David Mertz writes:
I don't think it can be a SyntaxError because you can't always know that mylist is a builtin list. list.__getitem__(self, ...) gets called and it tells you "TypeError: __getitem__() takes no keyword arguments", as it does now. (Just for the record. I bet you realized that within nanoseconds of hitting "send". ;-) Paul Moore writes:
True, but not really fair under my assumption that `x[i, default=d]` becomes the accepted way of doing it generally. dict.get would be a legacy in that case. How do you folks feel about list.get and tuple.get (I expect many folks will think differently about these because tuples are immutable), and more generally Sequence.get?
confusing, and unlike any other language I'm familiar with (all IMO, if that's not obvious...).
All this opposition is interesting. For reasons similar to those expressed by MAL for sequences I rarely use dict.get, let alone want list.get. To me it usually just feels wrong to be getting something that might not be there, and then leaving it that way. I use defaultdict much more than I use dict.get. Regarding the statement that it's "unlike any other language," I suspect that's true; the only possible exception would be Lisps, but I think the argument that they don't have true indexing ends it. Partly by analogy to Lisp, I think of indexing as an optimized case of function call (which is how it's implemented in Python! That's just an amusing observation, not intended seriously.) The square brackets basically say "you can expect this operation to be O[1], almost always with small c" for appropriate values of "small". So this syntax doesn't bother me, given that functions support arbitrary keywords. I guess all this makes me an extremist. Doesn't stop me from liking the Python you prefer just as well, though. :-)
On Fri, 28 Aug 2020 at 09:30, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
So one of [,default=] and dict.get is redundant. Why make the existing way the redundant one? What's the actual benefit? Particularly as *other* uses of keywords in indexes may well want to allow general keywords (Pandas named columns, for example) so we can't reasonably reserve "default" to mean a default value in the general case. Conversely, having a get method (if it exists) behave like the stdlib dict.get seems a reasonable application of duck typing.
I'm indifferent to list.get. I've never felt any need for it, and it's easy to implement if someone wants it (as you note in another mail). Making it a list method just means it doesn't work as well as a standalone function (in the sense that it's not going to support other types via duck typing). I see absolutely no point in tuple.get, it seems like the only reasons for it would be over-generalisation, or as a workaround for the fact that methods don't duck-type (x.get(i, default) where you don't know if x is a list or tuple - a get *function* would of course handle this perfectly via duck typing).
I use dict.get a lot - as MAL says, not being sure if a particular key is present in a dict is perfectly normal, in contrast to not being sure if an index is valid for a list. Consider that "key in dict" is a language feature, but there's no "index in list" syntax...
Lisp is an exception to most things :-)
I guess all this makes me an extremist. Doesn't stop me from liking the Python you prefer just as well, though. :-)
:-) Paul
Thanks for the comments! Some food for thought. One comment: Paul Moore writes:
On Fri, 28 Aug 2020 at 09:30, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Consistency with practice in other subscriptables. That makes it easier to learn the language as a whole. But it seems that most people are +/- 0 on .get, and -1 on default=, so .get is much more likely to become the consistent way to do things. Steve
On Fri, Aug 28, 2020 at 4:30 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I'm not nearly so clever :-). I was thinking, hypothetically, that it's possible for dict.__getitem__() to have code along the lines of: def __getitem__(self, index, **kws): if kws: raise SyntaxError("Keywords not allowed in dictionary index") # ... actual item getting ... I'm not saying that's a good idea per se. Hence my parenthetical. Probably any extra check on every dict access is a needless slowdown though. I only thought of it because `mydict[foo=bar]` now raises a SyntaxError, so raising something different would technically be a change in behavior. How do you folks feel about list.get and tuple.get (I expect many
folks will think differently about these because tuples are immutable), and more generally Sequence.get?
-1 on Sequence.get. -0.5 on tuple.get. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, Aug 27, 2020 at 8:24 AM David Mertz <mertz@gnosis.cx> wrote:
I agree -- this is very much a feature for third party packages -- or *maybe* some future stdlib class, but the builtins are fine as they are. In fact, I don't think there's a single use of a tuple of indexes (meaning something other than an arbitrary single object) in the stdlib is there? I know I've only used that in numpy. -CHB -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Aug 27, 2020, 1:15 PM Christopher Barker
I don't know whether it is in the stdlib, but I sometimes use tuples as dict keys. E.g. mydict[('mertz', 'david')] = 3.1415 Even though I could omit them, I'd almost surely use the parens for that. And more likely it would be: name = ('mertz', 'david') mydict[name] = 3.1415 Conceptually, an "immutable collection" serves a different purpose than "a collection of axes", even if they work then same under the hood.
On Thu, Aug 27, 2020 at 1:29 PM David Mertz <mertz@gnosis.cx> wrote:
What about something like this: class Name(NamedTuple): first: str last: str d = NamedKeyDict(Named) d[first='david', last='mertz'] = 1_000_000 # dollars --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 1:45 PM David Mertz <mertz@gnosis.cx> wrote:
Just intended to be a dictionary that uses a named tuple type (or other type with the same named attributes available) as the keys. You initialize it by telling it what the type is, similar to defaultdict: dd = defaultdict(list) ...except the type is a NT-like class: class MyNamedTuple(NamedTuple): spam: str eggs: str nd = NamedKeyDict(MyNamedTuple) Now if you add a key using named arguments, it will call MyNamedTuple and supply them as kwargs: nd[spam="foo", eggs="bar"] = "baz" ...results in:
nd {MyNamedTuple(spam='foo', eggs='bar'): 'baz'}
--- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 11:01 AM Ricky Teachey <ricky@teachey.org> wrote:
Are you suggesting that the built in dict be extended to support this? I'm pretty sure Jonathan Fine did suggest that -- but I don't think that's a good idea myself. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Aug 27, 2020 at 2:14 PM Christopher Barker <pythonchb@gmail.com> wrote:
No not the built-in, but maybe as an addition to the collections module. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 2:27 PM Ricky Teachey <ricky@teachey.org> wrote:
Actually it just occurred to me-- why limit the idea to namedtuples? The collections module could have a dict that maps the *args and **kwargs in the [ ] to any arbitrary callable: class MapDict: def __init__(self, f, __dict=None, **kwargs): self.callable = f self._dict = {} __dict = {**__dict if __dict is not None else {}, **kwargs} self.update(__dict) def __getitem__(self, key, **kwargs): args = self.translate_key(key) return self._dict[f(*args, **kwargs)] def __getitem__(self, key, value, **kwargs): args = self.translate_key(key) self._dict[f(*args, **kwargs)] = value def __delitem__(self, key, **kwargs): args = self.translate_key(key) del self._dict[f(*args, **kwargs)] def update(self, d, **kwargs): d.update(kwargs) for k,v in d.items(): self[k] = v def translate_key(self, key): try: ikey = iter(key) except TypeError: ikey = iter((key,)) return (*ikey,)
--- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 10:24 AM David Mertz <mertz@gnosis.cx> wrote:
Exactly -- that's what I meant by "an arbitrary single object" -- as far as dict is concerned, that's simply a hashable object. period, the dict code doesn't parse it out in any way at all.
Even though I could omit them, I'd almost surely use the parens for that. And more likely it would be:
Me too -- I always use the parens -- it never even dawned on me until this thread that I could omit them :-) But numpy on the other hand, when you pass it: arr[i, j] it specifically parses out that tuple, and uses i for the zeroth axis, and j for the first axis -- it is very much NOT an arbitrary tuple. And while you could do: indexes = (i, j) arr(indexes) That is very rarely done, because conceptually, you are passing two indexes, not one object. I've used Python and numpy (before that Numeric and numarray) for 20 years, and I only recently understood that [] always took only a single expression, and that the ability to pass multiple indexes was really only because you can make a tuple without parens, so that i, j is the same as (i, j). -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Aug 27, 2020 at 11:49:22PM +0900, Stephen J. Turnbull wrote:
I'm pretty sure that Guido said that he doesn't want the builtin dict and list types to get keyword arguments. That might change in the future, but I'm assuming that if keyword subscripts are supported, they will be added for the sake of projects like pandas, numpy, and xarray, not because there are plans to change the builtins.
It might have been the natural way to do it in 1991, if keyword subscripts existed back then. But they didn't, so now, in 2020, the natural way to do this in a dict is with the get method. That's not going to go away.
*shrug* If keyword subscripts are allowed; and if they become really popular outside of pandas etc; and people start using the `mapping[key, default=0]` idiom in third party mappings; and it becomes a defacto standard for mappings outside of the stdlib; and people want builtin dict to support the same idiom; then there might be some pressure on the core devs to add it to dict. But until such time, I wouldn't expect dicts to change. So I think this issue is a distraction. Unless the senior core devs like Guido, Brett, Nick, Serhiy, Raymond, etc (my apologies to anyone I missed) say that they're intending to add keywords to dict subscripting, I wouldn't expect the get method to become obsolete. I think the problem with list.get is that there are already a plethora of ways to do it: - Look Before You Leap: # assumes index is non-negative default if len(L) < index else L[index] - Easier to Ask Forgiveness than Permission: try: L[index] except IndexError: default - Slicing: # likewise assumes index is non-negative (L[index:index+1] or [default])[0] - any of which can be put into a helper function get(L, index, default) (Maybe add this to the operator module?) So unless this happens to scratch somebody's personal itch, I think you're going to run up against the problem that while this is useful, it's maybe not useful enough to bother. Personally, I ever so slightly lean towards the "if it already existed, I'd definitely use it, but I don't care enough to fight for it" position. I'm running out of energy to care about things I actually care about, and this is so far down the list that I'm honestly not sure why I'm even writing this :-) Probably as an avoidance tactic to avoid doing other more important things *wink* -- Steve
I just want to make clear that safe navegator is enough to deal with this. Now that this is clear, the use case is almost always the same. I received some json as response and want to extract a nested value. The way I'm doing this today is overloading an infix operator (I'm using
) to emulate safe navegator so I do
value = data >> get('foo') >> get(0) >> get('bar') if value: do something This get can be found here https://github.com/dhilst/geckones_twitter_bots/blob/e6aefe036d30f8483502605... I started with array implementation them extended to object too. I usually has something like this on all my projects that need to extract data from requests, or a function that receives a dotted path, anyway.. dict.get composes so nicely so why we don't have list.get too? Well, the safe navigator can solve this as it will compose as well as any adhoc solution with the advantage of short circuit and also it's brevity so ... For my use case, at last, safe navigator (or null coalescing) will fit better than any list.get can do. Regards Em ter, 30 de jun de 2020 18:53, Juancarlo Añez <apalala@gmail.com> escreveu:
On 2020-06-30 23:26, Daniel. wrote:
[snip] I'd probably just write a function for it: def get(data, default, *indexes): try: for index in indexes: data = data[index] except (IndexError, KeyError): return default value = get(data, None, 'foo', 0, 'bar') if value is not None: do something
toolz.get_in is designed for exactly this. https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.get_in The whole library is great! Alex On Wed, 1 Jul 2020 at 8:03 am, MRAB <python@mrabarnett.plus.com> wrote:
Very similar things could be said about dict.get too, no? It's easy to write your own function that does the same thing or to catch an exception. On the other hand, it seems far more likely to miss keys in a dictionary than it is to repeatedly mistake indices in a list.
Which would be the use cases for this feature?
I can't think of one.
I've never missed this feature either. But I would also be interested in hearing a dream use case. - DLD -- David Lowry-Duda <david@lowryduda.com> <davidlowryduda.com>
On Tue, Jun 30, 2020 at 8:55 PM David Lowry-Duda <david@lowryduda.com> wrote:
On the other hand, it seems far more likely to miss keys in a dictionary than it is to repeatedly mistake indices in a list.
exactly -- dict keys are arbitrary, and it's pretty common to store "sparse" data by simply leaving out the key if there's nothing interesting attached to it. But for Sequences, the only time you get an index error is if you are indexing beyond the length of the list, and the only case I can think of for get() would be for a maybe-empty list. otherwise, would I really want the same thing, and no error, for ANY index larger than the length of the list? Rather, if there's a specific index we want, we want it to be there, and if not, then we are iterating over it, which handles any length (including zero) just fine. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Jul 1, 2020 at 6:23 AM Christopher Barker <pythonchb@gmail.com> wrote:
dict.get is definitely much more useful. But list.get is still useful reasonably often. You can see quite a bit of demand for the method here: https://stackoverflow.com/questions/5125619/why-doesnt-list-have-safe-get-me... Below are some use cases I found and how they could be refactored with .get. Apart from the first one, they were all found simply by grepping for `except IndexError`. It's pretty easy to pick obvious use cases out from there. In some cases readability suffers a bit and using .get might not be the best idea, but I've included them anyway so that we can see what over-eager usage of the method might look like. ``` _pytest/_code/source.py if insert_index >= len(values): end = None else: end = values[insert_index] end = values.get(insert_index) -------------------- prompt_toolkit/layout/layout.py try: return self._stack[-2].content except IndexError: return self._stack[-1].content return self._stack.get(-2, self._stack[-1]).content -------------------- prompt_toolkit/buffer.py try: word = words[state.n] except IndexError: word = "" word = words.get(state.n, "") -------------------- prompt_toolkit/widgets/menus.py try: selected_item = self.selected_menu[level + 1] except IndexError: selected_item = -1 selected_item = self.selected_menu.get(level + 1, -1) -------------------- prompt_toolkit/document.py try: return self.text[self.cursor_position + offset] except IndexError: return "" return self.text.get(self.cursor_position + offset, "") -------------------- prompt_toolkit/contrib/regular_languages/lexer.py try: return lines[lineno] except IndexError: return [] return lines.get(lineno, []) -------------------- pip/_internal/cli/autocompletion.py try: subcommand_name = [w for w in cwords if w in subcommands][0] except IndexError: subcommand_name = None subcommand_name = [w for w in cwords if w in subcommands].get(0) -------------------- pip/_internal/cli/autocompletion.py try: current = cwords[cword - 1] except IndexError: current = '' current = cwords.get(cword - 1, '') -------------------- pip/_internal/configuration.py try: return self._get_parser_to_modify()[0] except IndexError: return None return self._get_parser_to_modify().get(0) -------------------- pip/_vendor/distlib/resources.py try: result = self.index[i].startswith(path) except IndexError: result = False result = self.index.get(i, "").startswith(path) -------------------- stack_data/core.py try: return not self.lines[i - 1].strip() except IndexError: return False return not self.lines.get(i - 1, "not blank").strip() -------------------- parso/tree.py try: return self.parent.children[i + 1] except IndexError: return None return self.parent.children.get(i + 1) -------------------- ipython/IPython/core/completer.py try: return self.matches[state] except IndexError: return None return self.matches.get(state) -------------------- ipython/IPython/core/magics/basic.py mode = '' try: mode = parameter_s.split()[0][1:] except IndexError: pass mode = parameter_s.split().get(0, '')[1:] -------------------- ipython/IPython/core/tests/simpleerr.py try: mode = sys.argv[1] except IndexError: mode = 'div' mode = sys.argv.get(1, 'div') -------------------- ipython/IPython/utils/text.py try: tgt = parts[field] return tgt except IndexError: return "" return parts.get(field, "") -------------------- py/_xmlgen.py try: tag.parent = self.parents[-1] except IndexError: tag.parent = None tag.parent = self.parents.get(-1, None) -------------------- ipython/IPython/core/tests/tclass.py try: name = sys.argv[1] except IndexError: pass else: if name.startswith('C'): c = C(name) name = sys.argv.get(1, "") if name.startswith('C'): c = C(name) ```
Em qua, 1 de jul de 2020 00:56, David Lowry-Duda <david@lowryduda.com> escreveu:
The usecase is extracting deeply nested data from parsed json. Missing keys are something that happens on dicts, I want something that is able to extract data from parsed json, that eliminates the error handling noise. All this talk elucidates me that this is possible try: v = foo.bar['tar'][0]['zar'] except (LookupError, AttributeError): v = None would suffice in all cases,
On 2020-06-27 09:34, Daniel. wrote:
I would use this. I do something similar already, albeit as a set of classes that wrap around Python `dict` and `list` to provide the null-safe access. to_data(somedict).foo.bar[10] Specifically, I wrap `list` in `FlatList`, which will return `Null` (null-safe version of `None`) instead of raising and IndexError. This allows math on indexes without concern for corner cases, and makes window functions easier to write: | my_list = FlatList(my_list) | deltas = [] | for i, this_week in enumerate(my_list): | last_week = my_list[i-7] | deltas.append(this_week - last_week) by avoiding exception handling, code is simplified, and procedures simplify to functions. Instead of code that constructs a list; the code reads as a list definition: | deltas = [ | this_week - last_week | for my_list in [FlatList(my_list)] | for i, this_week in enumerate(my_list) | for last_week in [my_list[i-7]] | ]
please forgive me: where `for x in [y]` can be read as `x=y` (I am still hesitant to use `:=`)
There are detriments to using a set of null-safe objects: 1. data structures can be polluted with a combination of null-safe objects and regular Python structures 2. with null-safe code, there are more nulls and it seems EVERYTHING must be null-safe, including arithmetic (notice null-safe subtraction above) 3. runs slower; all the null-safe checks dominate my profiler time. I would be nice if Python had a series of null-safe operators. Until then, `list.get` would eliminate my extra object creation: | deltas = [ | this_week - (last_week or 0) | for i, this_week in enumerate(my_list) | for last_week in [my_list.get(i-7)] | ] Although we see additional null-checks required (as #2 states).
Em sex, 3 de jul de 2020 11:37, Kyle Lahnakoski <klahnakoski@mozilla.com> escreveu:
Yeah, I just get the same idea but using rshift to emulate safe access infix operator somedict >> get('foo') Since I can't add behavior to exiting classes in python, I resort to things like that to "emulate" method calls. You can think somelist >> get(0) as somelist.get(0) So I don't have to deal with an extra type in translation when sending data back to a third party library
Yeah I feel that try/catch add lots of noise to null present code. We always tend to abstract null handling in some way and each one come up with its own implementation. The list.get method has the advatage, as Alex said, of already being a common dialect with dicts. If dict.get didn't exist I probably wouldn't even asking for it, but every time that I need to handle json and I start with dict.get and hit an list I end up with an wrapper over IndexError. try/catch are nice for medium blocks for single lines they are too many noise, you have 3 lines of error handling for one of real logic. (But you can wrap that in a function). Yeah, then you have to shift the reading flow to that function. When you see dict.get, you know what's happening, it's just straight to the point. If we had the same method for list, our codes may look similar while solving almost the same problem (handling null). But since there isn't we create whole new abstractions that are completely different.
It all depends of you're propagating None or not. And not if your code is None aware or not. But yeah, none aware code are easier to follow and write, at last for me, because the missing value handling is abstrated. But still, I'm not talking about to add a method to propagate more Nones, no, even if this is what .get does. I'm asking for something that can be used in conjunction with dict.get to deal with nested data in an elegant and idiomatic form so we stop to ac-hoc abstracting IndexError and start to use a common idiom. Another simple use case that I use this is to have give simple scripts a default command line argument, like try: arg = sys.argv[1] except IndexError: arg = default Or if len(sys.argv) >= 2: arg = sys.argv[1] else: arg = default Which would be rewritten as arg = sys.argv.get(1, default) I'm hopeful that there are more people with other uses case out there that didn't see the thread yet
3. runs slower; all the null-safe checks dominate my profiler time.
I would be nice if Python had a series of null-safe operators. Until then, `list.get` would eliminate my extra object creation:
I would really love it too, I was expecting to drop this in favor of pep 505 but as Alex pointed, it wouldn't handle IndexError case, so even with null operators we will be still having to handle with a except block
Regards
I think I'm missing something. Daniel wants a `list.get` method with similar semantics to `dict.get`. So instead of writing: ``` try: x = lst[i] except IndexError: x = default ``` one could write `x = lst.get(i, default)`. How would you rewrite that with PEP 505? I've also wanted this a couple of times, although I can't remember the reason. It's not just about traversing incomplete nested data structures. I don't see much disadvantage since the analogy with `dict.get` would make it very easy to learn and understand, to the point that I've wondered why this doesn't exist already. On Sat, Jun 27, 2020 at 5:09 PM Guido van Rossum <guido@python.org> wrote:
Em ter., 30 de jun. de 2020 às 15:49, Alex Hall <alex.mojaki@gmail.com> escreveu:
With PEP 505 is not possible to have an arbitrary default as the result of null coalesced expression will be None if a None is found. The lst.get(i) would be rewritten as lst?[i], and composing dct.get('foo', []).get(0, []).get('bar') would be something like dct?['foo']?[0]?['bar'], from what I read, this is very terse and terse is good in most cases good in my opinion. It removes the error handling noise by replacing it by one char I'm repeating myself here, but again, the use case is to fetch deedly nested data. All this talk make me notice that IndexError, and KeyError are LookupError instances but AttributeError is not, still pep 505 have an elegant solution for all these three. Also it handles the None?.method() case, when something return an object or None and you want to call a method on this if is not None. I think we should put more effort on pep 505
-- “If you're going to try, go all the way. Otherwise, don't even start. ..." Charles Bukowski
On Thu, Jul 2, 2020 at 9:33 PM Daniel. <danielhilst@gmail.com> wrote:
No, that's not right. `dct?['foo']` translates to `dct['foo'] if dct is not None else None`, which is very different from `dct.get('foo')`. Similarly `lst?[i]` is for when `lst` might be None, not empty. All PEP 505 gives you in the case where keys might be missing or lists might be empty is that you can write `.get` without defaults, e.g. `dct.get('foo')?.get(0)?.get('bar')`. I would quite like to have PEP 505, but I don't know how to revive it. And I'm not surprised that it's deferred, there's generally reluctance to add new syntax, especially symbols. On the other hand, `list.get` seems very doable to me. It's not new syntax. It would be extremely easy to learn for anyone familiar with `dict.get`, which is pretty much essential knowledge. You'd probably have some people guessing it exists and using it correctly without even seeing it first in other code or documentation. I haven't seen anyone in this thread suggesting any cost or downside of adding the method, just people asking if it would be useful. I feel like I answered that question pretty thoroughly, then the thread went quiet.
Alex Hall writes:
Bottom line: new use cases or modified semantics that are more attractive to people who don't sympathize with the "in NOSQL objects often aren't there" use case. Some handwavy discussion below. Note that Nick Coghlan in the PEP 622 thread over on python-dev is proposing a very different use for '?' which seems likely to conflict. It wouldn't rule out syntax, but definitely would make it more difficult to use '?' (though probably not syntactically impossible) for this purpose.
Unfortunately, the people whose answers you really want to hear don't hang out here much any more. Steven d'A has remarkably comprehensive knowledge, but he's not as good at channeling Guido on what isn't yet as he is at explaining what is already. Not sure where Andrew Barnert has got to recently, but put him and Steven together and you get the best of both that's available here. Greg Ewing is another valuable source, but his design instincts often are quite different from Guido's. Lots of good ideas and discussion from these developers, but add all that up and you don't get Guido or Barry or Nick. Not necessarily *worse*, but different, and if you can't show that some feature requires a horrible hack or inordinate amounts of hard-to-get-right trickery, being different from Guido still connotes being un-Pythonic, and that's hard to get around. (Don't let that stop you from arguing with Guido -- among his other remarkable characteristics, he's been known to change his mind and accept features he long opposed. :-) I don't suggest that you try to open up a thread on python-dev where the folks who dropped the acid in the Kool-Aid do show up frequently. All of the reasons you and others give for wanting this are well- known, and they just didn't do it. Until you've got a powerful new argument or different, more attractive semantics, you'll just get shooed back here. I can say that it's not just the high cost of a symbol that tabled that PEP. Chained accesses like "dct['foo'][0]['bar'] are just expressions, but among the old-timers there's a deep suspicion of the "fluent" style of programming, chaining method calls by returning self. This isn't the same thing, but "null coalescing" operators (I prefer "null propagating" but "coalescing" seems to be the preferred term among proponents) has something of the same flavor, albeit only for None. It necessarily privileges None[1], guaranteeing ambiguity of that value, especially in a context like parsing JSON, where JSON null is typically translated to Python None. There was also a concern that null coalescing encourages hard to debug, unmaintainable programming. I don't think anybody provided hard evidence on that count, but introducing an operator specifically to *postpone* detection of exceptional conditions just doesn't sit well with a lot of people. It confuses things that *should* be there but aren't, with things that *might* be there but aren't, and with None when it's actually there. Postponing error detection can be done with 'try', but try doesn't confuse those things.[2] The principle of "consenting adults" means that possibility of misuse won't kill an otherwise good idea, but the question "isn't there a more Pythonic, explicit semantics that doesn't allow errors to propagate?" is one way to get an idea tabled. Finally, once you compare null-coalescing operators with 'try', it's clear that 'try' is far more powerful as well as sufficient for this application, and it's just not clear that null-coalescing is sufficiently special a special case to deserve syntax. Don't @ me. That is, I don't have a horse in this race (except that it would be syntax I personally am unlikely to ever use, and probably won't even see very often). I'm just trying to summarize some of the themes of the discussion FYI. Footnotes: [1] Addition of a special NULL object was proposed, IIRC, but I forget why that was considered a bad idea. [2] On the other hand, ':=' got in even though that is what '=' is for. That's a precedent that PEP 505 didn't have, turning a statement into an expression.
On Fri, Jul 3, 2020 at 5:36 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Thanks Stephen, it's nice to get some idea of what happened to PEP 505. It would be great if someone could put an official summary in the PEP, like a "Deferral notice". However it feels like you misread my email. You quote me saying this:
but then apparently go on to talk about PEP 505. At that point I was talking about the low cost of a `list.get` method, in contrast to PEP 505.
On Thu, Jul 2, 2020 at 10:33 PM Alex Hall <alex.mojaki@gmail.com> wrote:
I just had a coworker ask if there was something akin to `list.get(0)`, so I'd like to try to revive this. I propose that: 1. There is basically no cost in terms of mental capacity, learnability, or API bloat in adding list.get because of the similarity to dict.get. 2. There are plenty of times it would be useful, as seen in https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC... 3. If the above two points are true, we should go ahead and add this. I think that the discussion here simply fizzled away because: 1. People got distracted by talking about PEP 505 which really isn't very relevant and would solve a different problem. 2. There are no major objections, so there isn't much left to talk about, which seems like a silly way for a proposal to die. The only decent objection I saw was skepticism about valid and frequent use cases but once I answered that no one pursued the matter.
I just came across this again while implementing an parser I would like to compare stack elements as if stack[-3] == x and stack[-2] == y and stack[-1] == z and somewere below elif stack[-1] == t I had to spread `len(stack)` in a lot of places. People said about the length of a list is usually known, but when you use it as a stack is the oposit. Em ter, 25 de ago de 2020 09:44, Alex Hall <alex.mojaki@gmail.com> escreveu:
Here is one example added in stdlib https://github.com/python/cpython/pull/14813/files It's checking the truthness of value before calling value[0], this is a perfect match for value.get(0) Here is another, just the same use case, https://github.com/python/cpython/pull/17515/files When you deal with parsing problems yielding sequence of tokens are pretty common and knowing how much tokens are there is a matter of input. When you deal with recursives problems using a stack too.. so there are use cases out there and it's a tiny change Em ter, 25 de ago de 2020 09:52, Daniel. <danielhilst@gmail.com> escreveu:
I mean you could write these as: if stack[-3:] == [x, y, z] and elif stack[-1:] == [t] But plenty of use cases still exist ( https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC...) and I think we shouldn't need to keep talking about them. On Tue, Aug 25, 2020 at 2:54 PM Daniel. <danielhilst@gmail.com> wrote:
On 2020-08-25 13:52, Daniel. wrote:
if stack[-3 : ] == [x, y, z]:
and somewere below
elif stack[-1] == t
elif stack[-1 : ] == [t]:
I had to spread `len(stack)` in a lot of places.
You can use slicing to reduce the number of `len(stack)`.
People said about the length of a list is usually known, but when you use it as a stack is the oposit.
[snip]
On Tue, Aug 25, 2020 at 5:50 AM Alex Hall <alex.mojaki@gmail.com> wrote:
A lot of ideas fizzle away on this list. The fact is that it's a lot of work to get a new feature accepted into Python, even if it's not very controversial -- it's the classing "scratching an itch" problem -- someone needs to find it itchy enough to do a LOT of scratching -- any idea needs a champion that will carry it though. This one is easier than most because it's pretty much a do we or don't we question, with the spelling semantics, all pretty much already decided. Though I haven't quite seen it said explicitly -- is this proposed to be added to the Sequence ABC? If so, there is a potential problem -- the ABC's "namespace" is not reserved, so if you add .get(), then any code out in the while that is already using .get in a custom sequence could potentially break. See the discussion about the idea of adding a .view() method to Sequences -- same issue. Even of you think it's a good idea, it's still hard to add a new (non-dunder) method to an ABC. -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Tue, Aug 25, 2020 at 7:23 PM Christopher Barker <pythonchb@gmail.com> wrote:
I think if it's a potential problem to add it to the ABC then let's just defer that and only add it to the built in sequences. But I'm not clear on what the problem is. If you have some code like this: ``` class MySequence(Sequence): def get(self, i): ... MySequence().get(3) ``` and then add .get to the Sequence ABC, the existing code will not be immediately broken because the custom MySequence.get overrides the ABC method so everything behaves as before. A problem that I can see eventually arising is if someone writes some generic code for Sequences using .get, MySequence won't work with it. That won't happen immediately and I'm not sure if I'd call it a break in compatibility. But I can see that it is a problem. Maybe if we build up enough methods to add to the collection ABCs (Sequence.view, Sequence.get, Set.intersection/union/etc, Mapping.__[i]or__) we can justify adding them all at once in Python 4 or in a new module collections.abc.v2 which has subclasses of the originals.
On Tue, Aug 25, 2020 at 10:51 AM Alex Hall <alex.mojaki@gmail.com> wrote:
On Tue, Aug 25, 2020 at 7:23 PM Christopher Barker
I think if it's a potential problem to add it to the ABC then let's just
defer that and only add it to the built in sequences.
Well yes, that’s a fine solution. Though an advantage of adding it into the ABC is that it could be a mixin, and all Sequences would them get it. But I'm not clear on what the problem is. If you have some code like this:
Yes, but then the third party object that was a Sequence may no longer be one. Maybe if we build up enough methods to add to the collection ABCs
(Sequence.view, Sequence.get, Set.intersection/union/etc, Mapping.__[i]or__) we can justify adding them all at once in Python 4
Careful now! No one wants to repeat the py2/3 transition! or in a new module collections.abc.v2 which has subclasses of the originals.
Maybe, but the trick is: how do you “build up” thus stuff before you have a place to put it? And what’s the point of a place to put new stuff, when there isn’t any new stuff? -CHB
-- Christopher Barker, PhD
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
As a point of interest, is get() considered an official part of the mapping protocol, or just nice-to-have? The docs don't seem to be very clear about that. There used to be some tables listing the methods making up the core sequence and mapping protocols, but I can't find them now. Have they been removed? Are the ABCs now the definitions of the protocols? -- Greg
On Wed, Aug 26, 2020 at 1:47 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I would also like to see an official answer to this. Are the mixin methods of the ABCs considered part of their contract? Is it documented somewhere? I think it should be. On the one hand it sort of feels like the contract is just the abstract methods that users are supposed to implement themselves, and the mixin methods are just a convenient benefit of subclassing. It's hard to say where that feeling comes from, but I think the name "mixin method" is part of it. I'm glad to see I'm not the only one that gets this feeling. On the other hand, if I have a variable of type Mapping, I'm usually going to assume it has a .get method that behaves like dict. Most tools will assume that too. If it doesn't, I'll be quite surprised and probably annoyed.
The mixins are part of the contract. You are free to override them, their implementation is just a helpful,default. FWIW I don’t think we should add a .get() method for Sequence or for list. It reeks of hyper-correctness. On Wed, Aug 26, 2020 at 01:33 Alex Hall <alex.mojaki@gmail.com> wrote:
On 27/08/20 2:32 am, Guido van Rossum wrote:
The mixins are part of the contract. You are free to override them, their implementation is just a helpful,default.
That doesn't really answer my question. To put it another way: If someone wanted to implement an object that obeys the mapping protocol, but without inheriting from the Mapping ABC, would they be obliged to provide a get() method? -- Greg
Yes. That's what I meant by "the mixins are part of the contract", sorry. On Wed, Aug 26, 2020 at 8:45 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Aug 26, 2020 at 4:32 PM Guido van Rossum <guido@python.org> wrote:
FWIW I don’t think we should add a .get() method for Sequence or for list. It reeks of hyper-correctness.
What do you mean by this? I want the method because it's useful and convenient, not because it's 'correct'. I don't think that lists are 'wrong' right now.
But for your convenience you are proposing a problematic change (see posts by others). Maybe I meant overgeneralization instead. On Wed, Aug 26, 2020 at 10:20 AM Alex Hall <alex.mojaki@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Aug 26, 2020 at 7:30 PM Guido van Rossum <guido@python.org> wrote:
But for your convenience you are proposing a problematic change (see posts by others).
I think we've established it's problematic to add it to the Sequence ABC, but in terms of only adding it to specific sequences like lists, I don't know what other posts you're referring to. Why is it problematic?
Maybe I meant overgeneralization instead.
Again, the goal isn't to generalize. I'm not trying to make lists look like dicts. It's not even possible to get close - `if i in x: print(x[i])` does very different things for lists and dicts.
On Wed, Aug 26, 2020 at 10:45 AM Alex Hall <alex.mojaki@gmail.com> wrote:
It would break duck typing -- if you check for IndexError you can accept pretty much any sequence. But if you use .get() you can only work with lists. This is why we have ABCs in the first place (and static checking). So that you can't add it to Sequence is already a smell. If it's in the Mapping protocol, why isn't it in the Sequence protocol? There's also the slippery-slope argument brought up earlier -- should it be added to tuple? To range? What should it do for a negative index? I also think it negatively affects readability. Today when I see `foo.get(bar)` I assume that (unless the code is full of tricks) foo is a dict or Mapping, and bar a key. But if list also grows a .get() method I have to assume in my head that foo is a list and then bar an int, and I basically have to read the rest of the code twice, once assuming foo is a Mapping, and once assuming it's a list.
The generalization I am talking about here is the assumption that because list and dict both support x[y], and dict also has x.get(y) which returns a default instead of raising, then it would be a good idea to add the latter also to list. And I judge it an overgeneralization because there's more to it than meets the eye. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Wed, Aug 26, 2020 at 7:56 PM Guido van Rossum <guido@python.org> wrote:
This is all just a general feature of ABCs. They were written once and frozen in place and no new methods can be added to them because it would break compatibility. It's not specific to this proposal. | and |= operators were added to dict but not to Mapping/MutableMapping for the same reason, even though the same operators exist with a similar meaning in the Set/MutableSet protocols. So there's a clear precedent for this. It's a little unfortunate that we can't put this in the Sequence ABC, but it's not a big deal. The vast majority of the time I'm interfacing with my own code and so I know that I'm working with a list, not some mysterious Sequence. I don't think people are going to change their public facing function signatures from `def foo(x: Sequence)` to `(x: list)` just so that they can use `x.get`. Even if they do, it suggests that they think `.get` is really useful, and callers can just convert their argument to a list first. Similarly adding the union operator to dict is fine because people usually use dicts, not generic Mappings. Besides, list already defines a bunch of stuff not found in Sequence or MutableSequence: copy, sort, `<`, `+`, and `*`. What's one more?
There's also the slippery-slope argument brought up earlier -- should it be added to tuple? To range?
I think Steven D'Aprano asked this. I don't see it as an argument against or a concern about a slippery slope, just a question. I think it'd be nice to have it on lots of sequences, but it's OK if it's not. Neither scenario strikes me as problematic. The union operator was also added to defaultdict, OrderedDict, and ChainMap. Was this a problem?
What should it do for a negative index?
I think most people expect that if `lst[i]` returns a value then `lst.get(i)` should return the same value, therefore it should return a default only when `lst[i]` fails. Sometimes you specifically want `lst.get(-1)` - there's a couple examples of that in [my earlier post]( https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC...) and Daniel recently presented a case when he wanted that. This isn't the ideal behaviour for all use cases, but only as much as normal indexing isn't the ideal behaviour for all use cases. Sometimes I've had bugs because I forgot to account for negative indices, but I'm still very grateful to you that negative indices work. We can't make something equally convenient for all use cases, but that doesn't diminish the value of this method. You said yourself that the generic alternative for all sequences is to "check for IndexError", a sign that doing the equivalent of that is more intuitive.
That's already a poor assumption to make. `get` is an extremely generic method name that has lots of meanings in different classes.
The purpose of `foo.get(bar)` is to get a value, not to tell you that foo is a dict. If the rest of the code isn't able to tell you whether `foo` is a dict or a list, then either the code is poorly written (and .get is not to blame) or it doesn't matter because you're just getting individual values. `foo[bar]` doesn't differentiate between lists and dicts and this is generally considered a good thing, not a problem.
I'm not assuming that list should have .get because dict has it. I'm claiming list should have it because it would often be useful, based on [empirical evidence]( https://mail.python.org/archives/list/python-ideas@python.org/message/7W74OC... ). Anyway, thank you for this. I'm enjoying the discussion and I'm glad to hear actual reasons that can be argued about rather than the proposal just fizzling away.
On Wed, Aug 26, 2020 at 12:38 PM Alex Hall <alex.mojaki@gmail.com> wrote:
and the Set ABC doesn't have the set methods, like .union, .intersection ... (though I still don't get why that is ...)
This is something to be thinking about, however -- having the builtins extend the ABCs hasn't been a big deal, what with dynamic typing and all -- but as folks start using type hints, and static type checking more, it could be an issue. Honestly, I haven't delved into type checking much, so maybe I'm making this up, but what if you write a function that is type hinted to take a Mapping, and uses | or |=, and all the test code uses a built in dict, and all seems to be well. Then this code starts being more widely used, and someone runs it with a different Mapping object, and it passes the static type checking pass, and then, oops, a run-time failure. It makes me wonder of the utility of the ABCs is the builtins are extending them. Maybe we need a set of (at least for testing) builtins that are fully ABC conformant ? -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Aug 26, 2020 at 10:10 PM Christopher Barker <pythonchb@gmail.com> wrote:
This made me curious so I did an experiment, but with Set because I couldn't be bothered to worry about Python versions. This code: ``` import typing import collections import builtins def foo(t: typing.Set, c: collections.Set, b: builtins.set): print(t + t) print(t | t) print(t.union(t)) print(c + c) print(c | c) print(c.union(c)) print(b + b) print(b | b) print(b.union(b)) ``` complains about `+` (as it should) and `c.union` in both PyCharm and mypy, and nothing else. So typing.Set is based on the builtin and not the ABC. If you want to support any subclass of collections.Set, your type hint needs to be collections.Set. Using typing.Set will fail to show warnings when you use a builtin only method, and will show warnings when you try to assign an ABC set (e.g. I get a warning with `foo(t=<instance of collections.abc.Set>)`. The problem then is that `collections.abc.Set[int]` doesn't work, i.e. you need to use typing.Set if you want generics.
Use typing.AbstractSet for the ABC (there was an unfortunate naming conflict -- we have dict and Mapping, list and Sequence, but set and Set...) On Wed, Aug 26, 2020 at 2:16 PM Alex Hall <alex.mojaki@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Oh thanks, I had no idea that existed. I wanted to experiment with Mapping and unions but it's not working. I couldn't find any existing issues about this. Should I be opening an issue? If not then there's no need to explain what I'm doing wrong. ``` $ cat test.py print({} | {}) $ ./python test.py {} $ mypy --python-version 3.9 test.py test.py:1: error: Unsupported left operand type for | ("Dict[<nothing>, <nothing>]") Found 1 error in 1 file (checked 1 source file) $ mypy --version mypy 0.782 ``` On Wed, Aug 26, 2020 at 11:18 PM Guido van Rossum <guido@python.org> wrote:
I think this method would be useful for list itself (maybe tuple, but I care less about that). It certainly should not be part of the Sequence protocol. I have often programmed a pattern where I .append() elements to a list in a loop, and *expect* at least N things to make it in. In such case, getting a sentinel answer to mylist.get(N-1) would be convenient. Obviously, I've lived without that fine. But it's a little bit shorter and more expressive than: x = mylist[N-1] if len(mylist) >= N else None On Wed, Aug 26, 2020, 1:30 PM Guido van Rossum <guido@python.org> wrote:
On Tue, Aug 25, 2020 at 10:22:20AM -0700, Christopher Barker wrote:
This one is easier than most because it's pretty much a do we or don't we question, with the spelling semantics, all pretty much already decided.
Is it to be added to lists, or lists and tuples? How about range objects? range(1, 9, 2).get(100) Strings and bytes? "abc".get(100) Do we require 3rd party sequences to add this as well, in order to continue being called sequences?
Though I haven't quite seen it said explicitly -- is this proposed to be added to the Sequence ABC?
If so, do we require that it return None as the default, or are types permitted to return whatever missing value they see fit? -- Steve
On Wed, Aug 26, 2020 at 1:41 AM Steven D'Aprano <steve@pearwood.info> wrote:
I think at a minimum it should be in lists and tuples. I can't see any reason not to add it to range, strings, bytes, deque, array, etc. beyond implementation effort. I also think it's fine to leave them out at first.
Do we require 3rd party sequences to add this as well, in order to continue being called sequences?
I think this is pretty much the same question as the discussion about ABCs. It seems like the answer is "no", or at least "not yet".
This should be identical to Mapping.get. None isn't required to be the default missing value - callers can specify any value they want - but None is the default default, i.e. `def get(self, item, default=None)`. I don't know of any Mapping or mapping-like class that has a different default default. If such classes exist, then I guess None is just the default default default :)
Alex Hall writes:
If the "keyword arguments in __getitem__" feature is added, .get() is purely redundant. (Of course this thread would then become "make 'default' a standard keyword argument for mutable collections.") Since .get() doesn't seem to be terribly useful, and there is quite a bit of support for keyword arguments in __getitem__, I think this can be tabled pending resolution of that proposal. Steve
On Wed, Aug 26, 2020 at 5:00 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Is that something people want to do? Do people want to be able to write `my_dict[key, default=0]` instead of `my_dict.get(key, 0)`? What about `my_dict[key, default=None]` instead of `my_dict.get(key)`?
Alex Hall writes:
Of course they don't want to do that. Of course dict.get is going nowhere. Of course that's redundant. Of course if collections get a standard 'default' for __getitem__, many people *will* start writing `my_dict[key, default=None]`, if only because they don't read enough docs to know about dict.get.
On Thu, Aug 27, 2020 at 10:15 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
OK, I'll try again. Do people want collections to get a standard 'default' for `__getitem__`? I don't want mappings to grow a second way to do the same thing, and I don't want sequences to have a different way to do it from mappings.
An .get method wont hurt duck typing, it makes it better. Also most of the time we're working with plain dicts and lists not abtract classes. If you're typechecking for a Sequence you cant treat it as a list, and this is already true because there are methods in list that dont exists Sequence as Alex said. Also the idea was always for a list, tuples and ranges are okay, but tuples are immutable and ranges are constructed so it's not that useful, as it is for lists derived from some unknown input It's simple: def get(self, i, default=None): try: return self[i] except IndexError: return default That was the idea, just a safe wrapper arround IndexError. Because it's very common to need to catch this. I posted two bugs that happened because of uncaught IndexErrors. .get() enforces better coding, it's more readable, avoid distracting indentation blocks and flow jumping, the error handling is reduced and more explicity and more fluid, you can get a fine default if you want. Besides breaking Sequences contract on list, (which is already broken as Alex said, by .copy, < and other methods) I cant see any other problem for adding it to *lists* Em qui, 27 de ago de 2020 05:57, Alex Hall <alex.mojaki@gmail.com> escreveu:
On 27.08.2020 14:40, Daniel. wrote:
I disagree on the above assessment. I have had such a get() builtin in mxTools for more than 20 years now and found that I hardly ever used it: https://www.egenix.com/products/python/mxBase/mxTools/doc/#_Toc293606201 The reason is simple: unlike for dicts, where you often expect non-existing items (e.g. in keyword arguments, config files, etc.), having a missing index position in a list which you parse is almost always an error. Now, errors should be clearly marked and handled as such, hence having a try-except is the better coding strategy. For those cases, where a list can have a variable number of entries (e.g. optional arguments, file lists, etc.), code should clearly branch on list length and then determine the right strategy to fetch items. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Aug 27, 2020 at 3:09 PM M.-A. Lemburg <mal@egenix.com> wrote:
I'm copying my earlier post because apparently linking to it repeatedly has no effect and people keep on claiming that this method wouldn't be useful while not saying a word about the evidence I presented that it would be. dict.get is definitely much more useful. But list.get is still useful reasonably often. You can see quite a bit of demand for the method here: https://stackoverflow.com/questions/5125619/why-doesnt-list-have-safe-get-me... Below are some use cases I found and how they could be refactored with .get. Apart from the first one, they were all found simply by grepping for `except IndexError`. It's pretty easy to pick obvious use cases out from there. In some cases readability suffers a bit and using .get might not be the best idea, but I've included them anyway so that we can see what over-eager usage of the method might look like. ``` _pytest/_code/source.py if insert_index >= len(values): end = None else: end = values[insert_index] end = values.get(insert_index) -------------------- prompt_toolkit/layout/layout.py try: return self._stack[-2].content except IndexError: return self._stack[-1].content return self._stack.get(-2, self._stack[-1]).content -------------------- prompt_toolkit/buffer.py try: word = words[state.n] except IndexError: word = "" word = words.get(state.n, "") -------------------- prompt_toolkit/widgets/menus.py try: selected_item = self.selected_menu[level + 1] except IndexError: selected_item = -1 selected_item = self.selected_menu.get(level + 1, -1) -------------------- prompt_toolkit/document.py try: return self.text[self.cursor_position + offset] except IndexError: return "" return self.text.get(self.cursor_position + offset, "") -------------------- prompt_toolkit/contrib/regular_languages/lexer.py try: return lines[lineno] except IndexError: return [] return lines.get(lineno, []) -------------------- pip/_internal/cli/autocompletion.py try: subcommand_name = [w for w in cwords if w in subcommands][0] except IndexError: subcommand_name = None subcommand_name = [w for w in cwords if w in subcommands].get(0) -------------------- pip/_internal/cli/autocompletion.py try: current = cwords[cword - 1] except IndexError: current = '' current = cwords.get(cword - 1, '') -------------------- pip/_internal/configuration.py try: return self._get_parser_to_modify()[0] except IndexError: return None return self._get_parser_to_modify().get(0) -------------------- pip/_vendor/distlib/resources.py try: result = self.index[i].startswith(path) except IndexError: result = False result = self.index.get(i, "").startswith(path) -------------------- stack_data/core.py try: return not self.lines[i - 1].strip() except IndexError: return False return not self.lines.get(i - 1, "not blank").strip() -------------------- parso/tree.py try: return self.parent.children[i + 1] except IndexError: return None return self.parent.children.get(i + 1) -------------------- ipython/IPython/core/completer.py try: return self.matches[state] except IndexError: return None return self.matches.get(state) -------------------- ipython/IPython/core/magics/basic.py mode = '' try: mode = parameter_s.split()[0][1:] except IndexError: pass mode = parameter_s.split().get(0, '')[1:] -------------------- ipython/IPython/core/tests/simpleerr.py try: mode = sys.argv[1] except IndexError: mode = 'div' mode = sys.argv.get(1, 'div') -------------------- ipython/IPython/utils/text.py try: tgt = parts[field] return tgt except IndexError: return "" return parts.get(field, "") -------------------- py/_xmlgen.py try: tag.parent = self.parents[-1] except IndexError: tag.parent = None tag.parent = self.parents.get(-1, None) -------------------- ipython/IPython/core/tests/tclass.py try: name = sys.argv[1] except IndexError: pass else: if name.startswith('C'): c = C(name) name = sys.argv.get(1, "") if name.startswith('C'): c = C(name) ```
On 27.08.2020 15:17, Alex Hall wrote:
If I'm not mistaken, all those listed cases fall under the second case I mentioned (branching on length). If your API allows for optional list entries, then it's better to branch on length (e.g. as in the pytest case). If this happens only in exceptional cases, use try-except (most other cases you listed). Both coding styles make the intent clear without doubt. As mentioned before, I had access to such a builtin for many years, but hardly ever chose to use it and instead opted for the more explicit branching on length. IMO, being able to write something with fewer key strokes is not always a good idea. Unless you write code only for yourself, it's usually better to emphasize on intent rather than brevity. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Aug 27, 2020 at 3:37 PM M.-A. Lemburg <mal@egenix.com> wrote:
All but the first example, i.e. all the uses of try-except, were found by grepping for `except IndexError`. That was just the easiest method I found to find examples. The fact that try-except was used in those cases does not necessarily mean that the index is missing "only in exceptional cases". Perhaps because you like to emphasise intent when you're coding you assume that "exceptional" is the intent when you see the keyword "except". It's not how I interpret these examples. I think truly exceptional cases would have a change in behaviour like showing a warning. All that's happening here is getting a default value, and using try-except is the easiest available syntax. Often the use case is that the list may be empty or you're at the edge of the list and you need to go one beyond. That's normal and expected, not exceptional. If list.get existed then probably it would have been used in many of these examples. But of course it didn't exist and I don't have a way of searching for custom get functions so nearly all the examples use except which you take to mean exceptional.
On 27.08.2020 16:01, Alex Hall wrote:
Could be that authors thought try-except to be the easier to use syntax, even when the length check is less effort to write, faster and clearer in stating that the missing items are expected. If those quoted code snippets really all refer to non-exceptional cases, it would be better to guide people to the length branching approach. And you could be right that some authors may have opted for list.get(), but the point of implementing such changes goes somewhat beyond "it reduced number of keystrokes". If it results in people writing less obvious code, it's usually not a good idea. Take e.g. the dict.get() interface. We should have not given it a default argument of default=None, since it's not obvious that mydict.get(x) will return None in case x is not part of mydict. Instead, it would have been better to require the default argument, so that you have to write mydict.get(x, None) or even mydict.get(x, default=None). That ship has sailed, but it should be warning to think twice about adding such apparently useful things. I had thought mx.Tools.get() would be useful at the time and it was easy to implement, along with several other additions. Many of them did not get much use. Others ended up in similar form in itertools or operator. The most useful addition was irange(), which I used a lot. Others had very similar needs and eventually enumerate() was added to the builtins. BTW: I also think that the analogy between dict.get() and list.get() is bit off... Asking a dict d of potentially any number of items for existence of a particular item x is somewhat different than asking a list l of a certain length i for the item at position i+1. The programmer should know that there is no item at position i+1. However, it's not obvious before trying that item x doesn't have a mapping in d by just looking at the object properties of d. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Aug 27, 2020 at 11:37 AM M.-A. Lemburg <mal@egenix.com> wrote:
Suppose we didn't have dict.get(). I would then probably write: val = mydict[key] if key in mydict else None Likewise, since we don't have list.get(), I would write: val = mylist[N] if len(mylist) >- N-1 else None Neither of those is impractical, but in both cases .get(key) seems to express the intent in a more obvious way. That said, it is trivial to write a get() function that does the same thing... and I've never bothered to do so. In fact, I could write a get() function that was polymorphic among mappings and sequences with very little work, which I've also never done. So I guess my case for the huge importance is undercut :-). -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On 27.08.2020 17:53, David Mertz wrote:
Really ? Let me try again :-) Getting an item from a bag of unknown items can easily fail, because you don't what's in the bag until you try to find the item. Trying to access an item from a non-existing bucket in an array will always fail. You know that in advance, since the bucket doesn't exist as per the array properties. You don't have to try finding it first. Not sure whether I'm getting the difference across. Grabbing into the void is different than grabbing into the dark :-) You might actually find something in the dark.
Well, I did a long time ago and hardly ever used it. For me, both are signs of "looks useful on the outside, but isn't in reality". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 27 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
M.-A. Lemburg writes:
On 27.08.2020 17:53, David Mertz wrote:
I think David is comparing the .get syntax to the if-else syntax. First of all, he got the if-else wrong, which is a sign that .get is easier to use, at least for writers and quite likely for readers. Second I agree that ".get" is easier to read and understand (especially with explicit default, as you advocate). It's even more stark if you're after a component of the list element: val = mylist[N if len(mylist) > N else default].content vs. val = mylist.get(N, default).content
But this isn't a C array. You don't know the length from the syntax. So you have to "do work" to discover the bucket isn't there before you access -- or you can delegate to the interpreter and catch IndexError. Given how precise IndexError is (it only catches integer arguments that fall outside the range of the array), I don't really see the argument for refusing to encapsulate the try in a .get method, or that its semantics differ from explicit bounds-checking. It seems to me that you're expressing a preference for "look before you leap" over "easier to ask forgiveness than permission" in the case of array indicies, which is a stylistic thing. It's a style I prefer in this case, but I can also see how other people would prefer a different style, one that makes sense to encapsulate in .get. Is there something more than stylistic preference that I'm missing?
On 28.08.2020 10:30, Stephen J. Turnbull wrote:
I was trying to explain why dict.get() is different than the suggested list.get(). dict.get() was added since the lookup is expensive and you want to avoid having to do this twice in the common case where the element does exist. It was not added as a way to hide away an exception, but instead to bypass having to generate this exception in the first place. dict.setdefault() has a similar motivation. list.get() merely safes you a line of code (or perhaps a few more depending on how you format things), hiding away an exception in case the requested index does not exist. If that's all you want, you're better off writing a helper which hides the exception for you. I argue that making it explicit that you're expecting two (or more) different list lengths in your code results in more intuitive and maintainable code, rather than catching IndexErrors (regardless of whether you hide them in a method, a helper, or handle them directly). So this is more than just style, it's about clarity of intent. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Aug 28 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Fri, Aug 28, 2020 at 4:30 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
It was an unintentional mistake, but Stephen is right it makes the point. I fixed above, both my thinko reversing the N vs. N-1, and also the simple typo of `>-` instead of `>=`. No language change is going to fix the fact the '-' and '=' keys are next to each other on my keyboard. I think the comparison (with right syntax and semantics) is fair. Whatever the ontological significance MAL notes about "reaching into the void" versus "reaching into the dark", the code I would actually write if dict.get() didn't exist is pretty much the above. And it looks a whole lot like the code I actually do write for possibly-missing index positions. As a side note, I don't really get why everyone else thinks a try/except is the most natural approach while a ternary seems more obvious to me for this situation. But it kinda connects to me liking list.get() better, I think... since "not enough items" doesn't seem as *exceptional* to me as it apparently does to some others. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On 8/28/20 8:24 AM, David Mertz wrote:
try/except is specifically looking for the error in question, index out of bounds. The ternary requires restating in other words the bounds limit. For instance, for a list of N items, the allowable subscripts are -N to N-1, and the sample ternary is only checking for the upper bound, not the lower, so fails an too small of an index (maybe you know you are only using positive indexes?) Missing this won't happen with try/except. -- Richard Damon
On 2020-08-28 at 08:41:04 -0400, Regarding "[Python-ideas] Re: What about having a .get(index, default) method for arrays like we have for dicts?," Richard Damon <Richard@Damon-Family.org> wrote:
Without more context, I've lost the forest for the trees. There are multiple ways to do this, and the "best" way likely depends on the context and not the readability, writability, maintainability, or expressivity of that exact line of code. Where did the index come from? How/when/where was the list constructed? Could this "problem" have been recognized and/or addressed sooner? What, exactly, are the conditions that cause the index being out of bounds at this point in the logic, and what are the semantics now that it's happened? Given the preceding logic, is an index that's too small (i.e. mathematically less than the opposite of the lenght of the list) even possible? I didn't find either of the examples (one using an if/else ternary and one using a hypothetical list.get method) particulary easy to read or particularly indicative of an obvious use case. And at that point, re-writing one [presumably correct] line of code into another won't make or break anything.
On Fri, 28 Aug 2020 at 13:26, David Mertz <mertz@gnosis.cx> wrote:
As a side note, I don't really get why everyone else thinks a try/except is the most natural approach while a ternary seems more obvious to me for this situation. But it kinda connects to me liking list.get() better, I think... since "not enough items" doesn't seem as *exceptional* to me as it apparently does to some others.
Possibly because in reality, people very rarely write code like that, so they are working out what they'd use "from scratch", rather than just quoting a well-known idiom that they are familiar with? Which to me says that this method isn't nearly as useful in practice as some people are claiming. Paul
Em qui, 27 de ago de 2020 10:37, M.-A. Lemburg <mal@egenix.com> escreveu:
In this case is pretty obvious the intent and this is an know abstraction. Avoid brevity will not make it cleaner. IMHO, explicitng index bounds and try/catch are distracting noise which hides intent instead. Also .get() would not prevent anybody from explicity checking bounds or catching index exceptions when appropriate
On Thu, Aug 27, 2020 at 03:17:24PM +0200, Alex Hall wrote:
Perhaps it's just that people don't think that the examples are compelling. Or they are burning out and finding it hard to care. Personally, I found your examples underwhelming because they're mostly repetitions of the same pattern. I'd find it more interesting if there were a larger variety of cases, not merely a larger number, and I'd find it *really* interesting if you found a couple of examples of bugs caused by the common idioms. E.g. the slicing and "branch of length" idioms are tricky to get right if the index is negative. Find a couple of bugs in publicly visible code because people forgot to take negative indices into account, and you might find that more people will sit up and take notice. -- Steve
On Thu, Aug 27, 2020 at 6:32 PM Steven D'Aprano <steve@pearwood.info> wrote:
Personally, I found your examples underwhelming because they're mostly repetitions of the same pattern.
That's surprising to me. When I see the same pattern over and over that's when I most want to refactor into a common function/method. The repetition is annoying and the solution is easy and obvious. If the cases were all different it'd be much more complicated and chances are there wouldn't be a simple function that covered all cases. Do you not usually refactor duplicated code? What *do* you refactor? I'd find it more interesting if there
were a larger variety of cases
This sort of sounds like the problem. The examples aren't meant to be interesting.
On Thu, Aug 27, 2020 at 07:10:38PM +0200, Alex Hall wrote:
The repetition doesn't go away just because you give it a name. It's just easier to manage.
Depends on what the code is. If it's as trivial as a try...except with a single operation inside each block, it's hardly worth the bother to add an extra level of indirection, an extra function, just to reduce what is effectively a two-liner (albeit one which is commonly spread over four lines) to a one-liner: # Before try: x = L[index] except IndexError: x = "" # After x = get(L, index, "") Save a line of code, replace it with one more function to read, document and test. Instead of having to read a two (four) line try...except block where everything is explicitly in front of my eyes, I have to read a fairly bland and generic-looking "get" function where I'm not sure *precisely* what it does witout reading the docs. Get what? Does it support negative indices? Can it fail? If I change L to a deque will it continue to work? I'm not *against* it. I think if lists already had the method I wouldn't object to its existence. I'm just finding it hard to care, and I think that's probably the main issue you are up against. This doesn't have a compelling benefit, it's quite marginal, at least in the use-case above. That's my opinion, others may disagree.
You're asking why people aren't interested in solving this issue, you wonder why few people are terribly interested in it, and now you're saying that its not supposed to be interesting. I think that explains the relative lack of interest :-) That's just my two cents though, others may feel differently. -- Steve
Alex Hall writes:
Do you not usually refactor duplicated code?
Taking "you" as generic: it depends.
What *do* you refactor?
Code is expressive. In cases where the duplication results in equivalent objects, I habitually refactor. When the process is important to express and the duplication is syntactic, I will prefer an explicit but idiomatic syntax repeated in unrelated contexts. Thus I generally wouldn't consider your list to be "duplicated code that cries out for refactoring." When the process is complex then I'd refactor. Detecting an out of bounds index is complex enough because of fenceposts and negative indicies, although I probably wouldn't notice the negative indicies until they bit me (and in that case wouldn't refactor until bitten).
"You [both] keep using that word but I do not think [Steven] means what you think it means." (pace, Inigo) I find the long list of examples with mostly identical syntax *unpersuasive* for two reasons. First, if you're going to bounds check array indicies explicitly (rather than allow IndexError to "bubble up"), we already have a facility for doing that precisely and accurately: catch IndexError. That catches out of bounds indicies and nothing else[1], and it handles negative indicies correctly. list.get can't be implemented much more efficiently than by using try ... except IndexError, and invoking it imposes attribute lookup and function call overhead so I guess you net approximately nothing on time, and a little bit of space. If you care so much about either time or space, you'd implement inline, something like: value = lst[index] if - (N := len(lst)) <= index < N else default try ... except IndexError is TOOWTDI now, it's as readable as .get and more readable than the inline version, so introducing list.get would be redundant. Second, I can see why people would want it (see my reply to MAL), I just don't want it myself, and implementing your own *correctly* as a function is trivial: def get(container, subscript, default): try: return container[subscript] except (IndexError, KeyError): return default It follows that I think dict.get is a YAGNI, at least since the introduction of KeyError, and definitely now that we can write y = mydict[x] if x in mydict else default Footnotes: [1] It turns out that slice bounds can't be out of bounds, they simply get truncated, in the degenerate case to the empty slice.
Em qui, 27 de ago de 2020 10:09, M.-A. Lemburg <mal@egenix.com> escreveu:
Almost always an error is not always an error and errors are not always exceptions, a missing index maybe something expected.
Branching has the same effect of try/catch, it will break code flow and make it more complex, so less readable. .get() is an nice abstraction to get something or return a default, well understood, it cant be less readable. I I came across this many times, many enought to come here to ask, and the use cases we're already showed, so I don't think, is useless would be a good argument. I understand that people disagreed on what is readable, but not that it rarely needed
Alex Hall writes:
OK, I'll try again. Do people want collections to get a standard 'default' for `__getitem__`?
I don't know.
I don't want mappings to grow a second way to do the same thing,
They're going to get one, is my reading of the "named indicies" thread. That is, it looks to me very likely that index notation (`x[i]`) is going to support keyword arguments (`x[i, k=v]`). If so, that's going to be a natural way to support a `default` argument. I imagine some people will choose it because they like it, and if there are enough people who do there will be pressure for it to be TOOWTDI.
and I don't want sequences to have a different way to do it from mappings.
I think you have to choose a second way for mappings to get, or have sequences do it differently, or have redundancy in sequence operations. If the index operator supports keyword arguments (I think it likely; YMMV), you either have to give dict a keyword argument or you have to give sequences a .get method. The former redundancy implied for dict can be justified by backward compatibility with dict and consistency with sequences. The latter is just an arbitrary consistency, and I doubt it will happen. That's just my guess; I have no power to make it go either way.
On Thu, Aug 27, 2020 at 10:51 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I support named indices. But I strongly oppose using them in list, tuple, or dict themselves. So `mylist[99, default=4]` would still be a syntax error (or maybe a different exception). I only want named indices for data structures where a clear and compelling use is demonstrated (like xarray). But allowing them syntactically allows third parties to play around in their own classes to see if they are helpful. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, 27 Aug 2020 at 16:21, David Mertz <mertz@gnosis.cx> wrote:
Agreed - allowing keywords in subscripts is OK (I'm personally ambivalent about the value of it, but if people want it, sure). But only for 3rd party classes. No-one has come up with a good use case for allowing them in existing builtin classes, and I'm -1 on doing so without a strong use case. Furthermore, I'd be very, very strongly opposed to having x[i, default=d] be a way of supplying a default result if i isn't present in *any* subscriptable class, and particularly in dict as an equivalent for x.get(i, d). It's redundant, confusing, and unlike any other language I'm familiar with (all IMO, if that's not obvious...). Paul
David Mertz writes:
I don't think it can be a SyntaxError because you can't always know that mylist is a builtin list. list.__getitem__(self, ...) gets called and it tells you "TypeError: __getitem__() takes no keyword arguments", as it does now. (Just for the record. I bet you realized that within nanoseconds of hitting "send". ;-) Paul Moore writes:
True, but not really fair under my assumption that `x[i, default=d]` becomes the accepted way of doing it generally. dict.get would be a legacy in that case. How do you folks feel about list.get and tuple.get (I expect many folks will think differently about these because tuples are immutable), and more generally Sequence.get?
confusing, and unlike any other language I'm familiar with (all IMO, if that's not obvious...).
All this opposition is interesting. For reasons similar to those expressed by MAL for sequences I rarely use dict.get, let alone want list.get. To me it usually just feels wrong to be getting something that might not be there, and then leaving it that way. I use defaultdict much more than I use dict.get. Regarding the statement that it's "unlike any other language," I suspect that's true; the only possible exception would be Lisps, but I think the argument that they don't have true indexing ends it. Partly by analogy to Lisp, I think of indexing as an optimized case of function call (which is how it's implemented in Python! That's just an amusing observation, not intended seriously.) The square brackets basically say "you can expect this operation to be O[1], almost always with small c" for appropriate values of "small". So this syntax doesn't bother me, given that functions support arbitrary keywords. I guess all this makes me an extremist. Doesn't stop me from liking the Python you prefer just as well, though. :-)
On Fri, 28 Aug 2020 at 09:30, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
So one of [,default=] and dict.get is redundant. Why make the existing way the redundant one? What's the actual benefit? Particularly as *other* uses of keywords in indexes may well want to allow general keywords (Pandas named columns, for example) so we can't reasonably reserve "default" to mean a default value in the general case. Conversely, having a get method (if it exists) behave like the stdlib dict.get seems a reasonable application of duck typing.
I'm indifferent to list.get. I've never felt any need for it, and it's easy to implement if someone wants it (as you note in another mail). Making it a list method just means it doesn't work as well as a standalone function (in the sense that it's not going to support other types via duck typing). I see absolutely no point in tuple.get, it seems like the only reasons for it would be over-generalisation, or as a workaround for the fact that methods don't duck-type (x.get(i, default) where you don't know if x is a list or tuple - a get *function* would of course handle this perfectly via duck typing).
I use dict.get a lot - as MAL says, not being sure if a particular key is present in a dict is perfectly normal, in contrast to not being sure if an index is valid for a list. Consider that "key in dict" is a language feature, but there's no "index in list" syntax...
Lisp is an exception to most things :-)
I guess all this makes me an extremist. Doesn't stop me from liking the Python you prefer just as well, though. :-)
:-) Paul
Thanks for the comments! Some food for thought. One comment: Paul Moore writes:
On Fri, 28 Aug 2020 at 09:30, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Consistency with practice in other subscriptables. That makes it easier to learn the language as a whole. But it seems that most people are +/- 0 on .get, and -1 on default=, so .get is much more likely to become the consistent way to do things. Steve
On Fri, Aug 28, 2020 at 4:30 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I'm not nearly so clever :-). I was thinking, hypothetically, that it's possible for dict.__getitem__() to have code along the lines of: def __getitem__(self, index, **kws): if kws: raise SyntaxError("Keywords not allowed in dictionary index") # ... actual item getting ... I'm not saying that's a good idea per se. Hence my parenthetical. Probably any extra check on every dict access is a needless slowdown though. I only thought of it because `mydict[foo=bar]` now raises a SyntaxError, so raising something different would technically be a change in behavior. How do you folks feel about list.get and tuple.get (I expect many
folks will think differently about these because tuples are immutable), and more generally Sequence.get?
-1 on Sequence.get. -0.5 on tuple.get. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Thu, Aug 27, 2020 at 8:24 AM David Mertz <mertz@gnosis.cx> wrote:
I agree -- this is very much a feature for third party packages -- or *maybe* some future stdlib class, but the builtins are fine as they are. In fact, I don't think there's a single use of a tuple of indexes (meaning something other than an arbitrary single object) in the stdlib is there? I know I've only used that in numpy. -CHB -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Aug 27, 2020, 1:15 PM Christopher Barker
I don't know whether it is in the stdlib, but I sometimes use tuples as dict keys. E.g. mydict[('mertz', 'david')] = 3.1415 Even though I could omit them, I'd almost surely use the parens for that. And more likely it would be: name = ('mertz', 'david') mydict[name] = 3.1415 Conceptually, an "immutable collection" serves a different purpose than "a collection of axes", even if they work then same under the hood.
On Thu, Aug 27, 2020 at 1:29 PM David Mertz <mertz@gnosis.cx> wrote:
What about something like this: class Name(NamedTuple): first: str last: str d = NamedKeyDict(Named) d[first='david', last='mertz'] = 1_000_000 # dollars --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 1:45 PM David Mertz <mertz@gnosis.cx> wrote:
Just intended to be a dictionary that uses a named tuple type (or other type with the same named attributes available) as the keys. You initialize it by telling it what the type is, similar to defaultdict: dd = defaultdict(list) ...except the type is a NT-like class: class MyNamedTuple(NamedTuple): spam: str eggs: str nd = NamedKeyDict(MyNamedTuple) Now if you add a key using named arguments, it will call MyNamedTuple and supply them as kwargs: nd[spam="foo", eggs="bar"] = "baz" ...results in:
nd {MyNamedTuple(spam='foo', eggs='bar'): 'baz'}
--- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 11:01 AM Ricky Teachey <ricky@teachey.org> wrote:
Are you suggesting that the built in dict be extended to support this? I'm pretty sure Jonathan Fine did suggest that -- but I don't think that's a good idea myself. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Aug 27, 2020 at 2:27 PM Ricky Teachey <ricky@teachey.org> wrote:
Actually it just occurred to me-- why limit the idea to namedtuples? The collections module could have a dict that maps the *args and **kwargs in the [ ] to any arbitrary callable: class MapDict: def __init__(self, f, __dict=None, **kwargs): self.callable = f self._dict = {} __dict = {**__dict if __dict is not None else {}, **kwargs} self.update(__dict) def __getitem__(self, key, **kwargs): args = self.translate_key(key) return self._dict[f(*args, **kwargs)] def __getitem__(self, key, value, **kwargs): args = self.translate_key(key) self._dict[f(*args, **kwargs)] = value def __delitem__(self, key, **kwargs): args = self.translate_key(key) del self._dict[f(*args, **kwargs)] def update(self, d, **kwargs): d.update(kwargs) for k,v in d.items(): self[k] = v def translate_key(self, key): try: ikey = iter(key) except TypeError: ikey = iter((key,)) return (*ikey,)
--- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Aug 27, 2020 at 10:24 AM David Mertz <mertz@gnosis.cx> wrote:
Exactly -- that's what I meant by "an arbitrary single object" -- as far as dict is concerned, that's simply a hashable object. period, the dict code doesn't parse it out in any way at all.
Even though I could omit them, I'd almost surely use the parens for that. And more likely it would be:
Me too -- I always use the parens -- it never even dawned on me until this thread that I could omit them :-) But numpy on the other hand, when you pass it: arr[i, j] it specifically parses out that tuple, and uses i for the zeroth axis, and j for the first axis -- it is very much NOT an arbitrary tuple. And while you could do: indexes = (i, j) arr(indexes) That is very rarely done, because conceptually, you are passing two indexes, not one object. I've used Python and numpy (before that Numeric and numarray) for 20 years, and I only recently understood that [] always took only a single expression, and that the ability to pass multiple indexes was really only because you can make a tuple without parens, so that i, j is the same as (i, j). -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Aug 27, 2020 at 11:49:22PM +0900, Stephen J. Turnbull wrote:
I'm pretty sure that Guido said that he doesn't want the builtin dict and list types to get keyword arguments. That might change in the future, but I'm assuming that if keyword subscripts are supported, they will be added for the sake of projects like pandas, numpy, and xarray, not because there are plans to change the builtins.
It might have been the natural way to do it in 1991, if keyword subscripts existed back then. But they didn't, so now, in 2020, the natural way to do this in a dict is with the get method. That's not going to go away.
*shrug* If keyword subscripts are allowed; and if they become really popular outside of pandas etc; and people start using the `mapping[key, default=0]` idiom in third party mappings; and it becomes a defacto standard for mappings outside of the stdlib; and people want builtin dict to support the same idiom; then there might be some pressure on the core devs to add it to dict. But until such time, I wouldn't expect dicts to change. So I think this issue is a distraction. Unless the senior core devs like Guido, Brett, Nick, Serhiy, Raymond, etc (my apologies to anyone I missed) say that they're intending to add keywords to dict subscripting, I wouldn't expect the get method to become obsolete. I think the problem with list.get is that there are already a plethora of ways to do it: - Look Before You Leap: # assumes index is non-negative default if len(L) < index else L[index] - Easier to Ask Forgiveness than Permission: try: L[index] except IndexError: default - Slicing: # likewise assumes index is non-negative (L[index:index+1] or [default])[0] - any of which can be put into a helper function get(L, index, default) (Maybe add this to the operator module?) So unless this happens to scratch somebody's personal itch, I think you're going to run up against the problem that while this is useful, it's maybe not useful enough to bother. Personally, I ever so slightly lean towards the "if it already existed, I'd definitely use it, but I don't care enough to fight for it" position. I'm running out of energy to care about things I actually care about, and this is so far down the list that I'm honestly not sure why I'm even writing this :-) Probably as an avoidance tactic to avoid doing other more important things *wink* -- Steve
I just want to make clear that safe navegator is enough to deal with this. Now that this is clear, the use case is almost always the same. I received some json as response and want to extract a nested value. The way I'm doing this today is overloading an infix operator (I'm using
) to emulate safe navegator so I do
value = data >> get('foo') >> get(0) >> get('bar') if value: do something This get can be found here https://github.com/dhilst/geckones_twitter_bots/blob/e6aefe036d30f8483502605... I started with array implementation them extended to object too. I usually has something like this on all my projects that need to extract data from requests, or a function that receives a dotted path, anyway.. dict.get composes so nicely so why we don't have list.get too? Well, the safe navigator can solve this as it will compose as well as any adhoc solution with the advantage of short circuit and also it's brevity so ... For my use case, at last, safe navigator (or null coalescing) will fit better than any list.get can do. Regards Em ter, 30 de jun de 2020 18:53, Juancarlo Añez <apalala@gmail.com> escreveu:
On 2020-06-30 23:26, Daniel. wrote:
[snip] I'd probably just write a function for it: def get(data, default, *indexes): try: for index in indexes: data = data[index] except (IndexError, KeyError): return default value = get(data, None, 'foo', 0, 'bar') if value is not None: do something
toolz.get_in is designed for exactly this. https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.get_in The whole library is great! Alex On Wed, 1 Jul 2020 at 8:03 am, MRAB <python@mrabarnett.plus.com> wrote:
Very similar things could be said about dict.get too, no? It's easy to write your own function that does the same thing or to catch an exception. On the other hand, it seems far more likely to miss keys in a dictionary than it is to repeatedly mistake indices in a list.
Which would be the use cases for this feature?
I can't think of one.
I've never missed this feature either. But I would also be interested in hearing a dream use case. - DLD -- David Lowry-Duda <david@lowryduda.com> <davidlowryduda.com>
On Tue, Jun 30, 2020 at 8:55 PM David Lowry-Duda <david@lowryduda.com> wrote:
On the other hand, it seems far more likely to miss keys in a dictionary than it is to repeatedly mistake indices in a list.
exactly -- dict keys are arbitrary, and it's pretty common to store "sparse" data by simply leaving out the key if there's nothing interesting attached to it. But for Sequences, the only time you get an index error is if you are indexing beyond the length of the list, and the only case I can think of for get() would be for a maybe-empty list. otherwise, would I really want the same thing, and no error, for ANY index larger than the length of the list? Rather, if there's a specific index we want, we want it to be there, and if not, then we are iterating over it, which handles any length (including zero) just fine. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Wed, Jul 1, 2020 at 6:23 AM Christopher Barker <pythonchb@gmail.com> wrote:
dict.get is definitely much more useful. But list.get is still useful reasonably often. You can see quite a bit of demand for the method here: https://stackoverflow.com/questions/5125619/why-doesnt-list-have-safe-get-me... Below are some use cases I found and how they could be refactored with .get. Apart from the first one, they were all found simply by grepping for `except IndexError`. It's pretty easy to pick obvious use cases out from there. In some cases readability suffers a bit and using .get might not be the best idea, but I've included them anyway so that we can see what over-eager usage of the method might look like. ``` _pytest/_code/source.py if insert_index >= len(values): end = None else: end = values[insert_index] end = values.get(insert_index) -------------------- prompt_toolkit/layout/layout.py try: return self._stack[-2].content except IndexError: return self._stack[-1].content return self._stack.get(-2, self._stack[-1]).content -------------------- prompt_toolkit/buffer.py try: word = words[state.n] except IndexError: word = "" word = words.get(state.n, "") -------------------- prompt_toolkit/widgets/menus.py try: selected_item = self.selected_menu[level + 1] except IndexError: selected_item = -1 selected_item = self.selected_menu.get(level + 1, -1) -------------------- prompt_toolkit/document.py try: return self.text[self.cursor_position + offset] except IndexError: return "" return self.text.get(self.cursor_position + offset, "") -------------------- prompt_toolkit/contrib/regular_languages/lexer.py try: return lines[lineno] except IndexError: return [] return lines.get(lineno, []) -------------------- pip/_internal/cli/autocompletion.py try: subcommand_name = [w for w in cwords if w in subcommands][0] except IndexError: subcommand_name = None subcommand_name = [w for w in cwords if w in subcommands].get(0) -------------------- pip/_internal/cli/autocompletion.py try: current = cwords[cword - 1] except IndexError: current = '' current = cwords.get(cword - 1, '') -------------------- pip/_internal/configuration.py try: return self._get_parser_to_modify()[0] except IndexError: return None return self._get_parser_to_modify().get(0) -------------------- pip/_vendor/distlib/resources.py try: result = self.index[i].startswith(path) except IndexError: result = False result = self.index.get(i, "").startswith(path) -------------------- stack_data/core.py try: return not self.lines[i - 1].strip() except IndexError: return False return not self.lines.get(i - 1, "not blank").strip() -------------------- parso/tree.py try: return self.parent.children[i + 1] except IndexError: return None return self.parent.children.get(i + 1) -------------------- ipython/IPython/core/completer.py try: return self.matches[state] except IndexError: return None return self.matches.get(state) -------------------- ipython/IPython/core/magics/basic.py mode = '' try: mode = parameter_s.split()[0][1:] except IndexError: pass mode = parameter_s.split().get(0, '')[1:] -------------------- ipython/IPython/core/tests/simpleerr.py try: mode = sys.argv[1] except IndexError: mode = 'div' mode = sys.argv.get(1, 'div') -------------------- ipython/IPython/utils/text.py try: tgt = parts[field] return tgt except IndexError: return "" return parts.get(field, "") -------------------- py/_xmlgen.py try: tag.parent = self.parents[-1] except IndexError: tag.parent = None tag.parent = self.parents.get(-1, None) -------------------- ipython/IPython/core/tests/tclass.py try: name = sys.argv[1] except IndexError: pass else: if name.startswith('C'): c = C(name) name = sys.argv.get(1, "") if name.startswith('C'): c = C(name) ```
Em qua, 1 de jul de 2020 00:56, David Lowry-Duda <david@lowryduda.com> escreveu:
The usecase is extracting deeply nested data from parsed json. Missing keys are something that happens on dicts, I want something that is able to extract data from parsed json, that eliminates the error handling noise. All this talk elucidates me that this is possible try: v = foo.bar['tar'][0]['zar'] except (LookupError, AttributeError): v = None would suffice in all cases,
On 2020-06-27 09:34, Daniel. wrote:
I would use this. I do something similar already, albeit as a set of classes that wrap around Python `dict` and `list` to provide the null-safe access. to_data(somedict).foo.bar[10] Specifically, I wrap `list` in `FlatList`, which will return `Null` (null-safe version of `None`) instead of raising and IndexError. This allows math on indexes without concern for corner cases, and makes window functions easier to write: | my_list = FlatList(my_list) | deltas = [] | for i, this_week in enumerate(my_list): | last_week = my_list[i-7] | deltas.append(this_week - last_week) by avoiding exception handling, code is simplified, and procedures simplify to functions. Instead of code that constructs a list; the code reads as a list definition: | deltas = [ | this_week - last_week | for my_list in [FlatList(my_list)] | for i, this_week in enumerate(my_list) | for last_week in [my_list[i-7]] | ]
please forgive me: where `for x in [y]` can be read as `x=y` (I am still hesitant to use `:=`)
There are detriments to using a set of null-safe objects: 1. data structures can be polluted with a combination of null-safe objects and regular Python structures 2. with null-safe code, there are more nulls and it seems EVERYTHING must be null-safe, including arithmetic (notice null-safe subtraction above) 3. runs slower; all the null-safe checks dominate my profiler time. I would be nice if Python had a series of null-safe operators. Until then, `list.get` would eliminate my extra object creation: | deltas = [ | this_week - (last_week or 0) | for i, this_week in enumerate(my_list) | for last_week in [my_list.get(i-7)] | ] Although we see additional null-checks required (as #2 states).
Em sex, 3 de jul de 2020 11:37, Kyle Lahnakoski <klahnakoski@mozilla.com> escreveu:
Yeah, I just get the same idea but using rshift to emulate safe access infix operator somedict >> get('foo') Since I can't add behavior to exiting classes in python, I resort to things like that to "emulate" method calls. You can think somelist >> get(0) as somelist.get(0) So I don't have to deal with an extra type in translation when sending data back to a third party library
Yeah I feel that try/catch add lots of noise to null present code. We always tend to abstract null handling in some way and each one come up with its own implementation. The list.get method has the advatage, as Alex said, of already being a common dialect with dicts. If dict.get didn't exist I probably wouldn't even asking for it, but every time that I need to handle json and I start with dict.get and hit an list I end up with an wrapper over IndexError. try/catch are nice for medium blocks for single lines they are too many noise, you have 3 lines of error handling for one of real logic. (But you can wrap that in a function). Yeah, then you have to shift the reading flow to that function. When you see dict.get, you know what's happening, it's just straight to the point. If we had the same method for list, our codes may look similar while solving almost the same problem (handling null). But since there isn't we create whole new abstractions that are completely different.
It all depends of you're propagating None or not. And not if your code is None aware or not. But yeah, none aware code are easier to follow and write, at last for me, because the missing value handling is abstrated. But still, I'm not talking about to add a method to propagate more Nones, no, even if this is what .get does. I'm asking for something that can be used in conjunction with dict.get to deal with nested data in an elegant and idiomatic form so we stop to ac-hoc abstracting IndexError and start to use a common idiom. Another simple use case that I use this is to have give simple scripts a default command line argument, like try: arg = sys.argv[1] except IndexError: arg = default Or if len(sys.argv) >= 2: arg = sys.argv[1] else: arg = default Which would be rewritten as arg = sys.argv.get(1, default) I'm hopeful that there are more people with other uses case out there that didn't see the thread yet
3. runs slower; all the null-safe checks dominate my profiler time.
I would be nice if Python had a series of null-safe operators. Until then, `list.get` would eliminate my extra object creation:
I would really love it too, I was expecting to drop this in favor of pep 505 but as Alex pointed, it wouldn't handle IndexError case, so even with null operators we will be still having to handle with a except block
Regards
participants (19)
-
2QdxY4RzWzUUiLuE＠potatochowder.com -
Alex Hall -
Alexander Hill -
Chris Angelico -
Christopher Barker -
Daniel. -
David Lowry-Duda -
David Mertz -
Greg Ewing -
Guido van Rossum -
Juancarlo Añez -
Kyle Lahnakoski -
M.-A. Lemburg -
MRAB -
Paul Moore -
Richard Damon -
Ricky Teachey -
Stephen J. Turnbull -
Steven D'Aprano