Add "default" kw argument to operator.itemgetter and operator.attrgetter
Hi everybody, Our PEP idea would be to purpose to add a global default value for itemgeet and attrgetter method. This was inspired from bug 14384 (https://bugs.python.org/issue14384); opened by Miki TEBEKA. For example, we could do: p1 = {'x': 43; 'y': 55} x, y, z = itemgetter('x', 'y', 'z', default=0)(values) print(x, y, z) 43, 55, 0 instead of: values = {'x': 43; 'y': 55} x = values.get('x', 0) y = values.get('y', 0) z = values.get('z', 0) print(x, y, z) 43, 55, 0 The goal is to have have concise code and improve consistency with getattr, attrgetter and itemgetter What are you thinking about this? MAILLOL Vincent GALODE Alexandre
On Wed, May 02, 2018 at 10:08:55AM +0200, Vincent Maillol wrote:
I'm sorry, I'm not sure I understand what you mean by a global default. My interpretation of that would be the ability to set a global variable which acts as default for *all* calls to itemgetter. itemdefault = "spam" f = itemgetter(2, 99) f('abcd') # => returns ('c', 'spam') If that's what you mean, I hate it :-) [...]
I wouldn't call that a *global* default. If the default has to be specified on each call, I think that's both acceptible and desirable. Bug 14384 (https://bugs.python.org/issue14384) suggests that it might be more desirable to specify the default when you call the getter function, not when you call the factory: # this f = itemgetter(2, 99) f('abcd', default='spam') # => returns ('c', 'spam') # rather than this f = itemgetter(2, 99, default='spam') f('abcd') # => returns ('c', 'spam') I could go either way. In fact, at the risk of making a more complicated API, I'm almost inclined to allow both forms... -- Steve
On Wed, May 02, 2018 at 10:28:33PM -0700, Raymond Hettinger wrote:
Intended by whom? I think you are being too dismissive of actual use-cases requested by actual users. Default values might not have been the primary use considered when the API was first invented, but the fact that people keep asking for this feature should tell us that at least some people have intended uses that are remaining unmet.
At some point, we're really better off just using a lambda.
Maybe I'm slow today, but I'm having trouble seeing how to write this as a lambda. The single index case would be simple using the proposed binding- expression from PEP 572: # replace INDEX and DEFAULT with the values you want lambda seq: s[0] if (s := seq[INDEX:INDEX+1]) else DEFAULT but without the binding-expression, I have no idea how to do it cleanly and correctly in a lambda. For what it is worth, on my computer, itemgetter with a single index is 20% faster than a lambda using seq[INDEX] (where INDEX is hard-coded in the lambda), and 60% faster than a lambda using seq[INDEX:INDEX+1][0]. According to the bug report linked to earlier, one of the intended uses for itemgetter is when performance matters. -- Steve
On May 2, 2018, at 11:32 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Intended by whom?
By me. I proposed itemgetter() in the first place. That rationale I gave convinced Guido and python-dev to accept it. I then wrote the code, docs, tests and have maintained it for over a decade. So, I have a pretty good idea of what it was intended for.
I think you are being too dismissive of actual use-cases requested by actual users.
Wow, I don't know what to do with this. Over the years, I've added a lot of things requested by users. I really don't like the tone you've struck and what you've implied about me as developer. That feels somewhat pushy and aggressive. Why not just give a +1 to things that are a good idea and -1 for things we're better off without -- no need for ad hominem comments about the person making the post rather than its content -- that feels somewhat disrespectful.
When I've seen the request in the past, it always alway "it might be nice if ..." but there were no legitimate use cases presented, just toy examples. Also, I'm concerned that about increasing the complexity of itemgetter() API to serve an occasional exotic use case rather that being easy to learn and remember for the common cases. Raymond
On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote:
Maybe I'm slow today, but I'm having trouble seeing how to write this as a lambda.
Yes, I was definitely having a "cannot brain, I have the dumb" day, because it is not that hard to write using lambda. See discussion here: https://mail.python.org/pipermail/python-list/2018-May/732795.html If anything, the problem is a plethora of choices, where it isn't clear which if any is the best way, or the One Obvious Way. -- Steve
At one time, lambda was the one obvious way. Later, partial, itemgetter, attrgetter, and methodcaller were added to express common patterns for key-functions and map(). If needed, the zoo of lambda alternatives could be further extended to add a rpartial() function that partials from the right. That would have helped with Miki's example. Instead of: get = attrgetter('foo', None) return get(args) or get(config) or get(env) He could've written: get = rpartial(getattr, 'foo', None) return get(args) or get(config) or get(env) If itemgetter and attrgetter only did a single lookup, a default might make sense. However, that doesn't fit well with multiple and/or chained lookups where are number of options are possible. (See https://bugs.python.org/issue14384#msg316222 for examples and alternatives). Raymond
Hi everybody,
Have you encountered situations yourself where this would make a difference ? I need to get multiple objects in several dicts using the same set of keys.
I wanted to use itemgetter to reduce the number of lines of code but I have mixed up getattr/dict.get that have default parameter with attrgetter/itemgetter.
At some point, we're really better off just using a lambda. I kept severals line with `.get('key', value)`, I find that is more readable.
I did it just to try it with itemgetter https://github.com/Maillol/cpython/compare/master...Add-default-parameter-to... I don't know if we add parameter `default` to itemgetter, getitem, attrgetter the API will become more complexe or more consistency, but I agree with you, it is an occasional use case and we can always use `my_dict.get`. 2018-05-07 6:07 GMT+02:00 Raymond Hettinger <raymond.hettinger@gmail.com>:
(1) On 7 May 2018 at 01:07, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
That's somewhat hybrid, it would behave like: lambda obj: getattr(obj, "foo", None), but one might expect a "reversed partial" to have its arguments reversed as well, like: lambda obj: getattr(obj, None, "foo"). (2) Why getattr can't accept keyword arguments?
Since partial/partialmethod can apply keyword arguments, allowing the "default=" keyword argument in getattr would be an alternative to the "rpartial". (3) At one time, lambda was the one obvious way. [...]
I really like lambdas, but there are some "closure gotchas" like:
for idx in indices: ... do_something(lambda seq: seq[idx])
If the lambda is stored somewhere to be called after the loop ends (or after its iteration), the seq[idx] would load the item with the last iterated index. This:
for idx in indices: ... do_something(itemgetter(idx))
would behave more like this instead:
for idx in indices: ... do_something((lambda i: (lambda seq: seq[i]))(idx))
Which I wouldn't call "the one obvious way". (4) If itemgetter and attrgetter only did a single lookup, a default might make
As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default. Perhaps something like:
-- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)
As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default.
+1 Perhaps something like:
The keywords suffixed by indice are uncommon. Maybe we can use dedicated object.
Another possibility is that itemgetter becomes an object with `add_default` method.
or simply default parameter should be a list
itemgetter(-1, 0, 2, ... default=["first", itemgetter.NoDefault, "third"])
2018-05-08 1:37 GMT+02:00 Danilo J. S. Bellini <danilo.bellini@gmail.com>:
On Tue, May 08, 2018 at 10:13:43AM +0200, Vincent Maillol wrote:
Sorry Vincent, but I think that's precisely the sort of overly complicated API that Raymond was worried about when he voted against this proposal. A single default value that applies when the item (or key) isn't present is easy to comprehend. But this extention is, I think, a case of an over-engineered, excessively intricate solution, and I doubt it solves any real problem. And I'm not even sure I know what it means. You have keyword parameters default0, default1, and then *default* with no suffix. Is that a typo for default2, or is it supposed to be a default default, the default to use when no default is specified?
I'm afraid there's no elegance to this design either. I've had many occasions where I want to get an item, falling back on a default if it is not present. With dicts this is common enough that we have dict.get(). It is less convenient with lists, and I'd like to see itemgetter support that case. But I cannot think of any case where I have needed or wanted to extract item 5 with "spam" as the default item 2 with no default item 3 with "eggs" as the default (for example), let alone that this is *so common* that I'd rather read the complicated documenation to work out how to use the function, rather than just write a short helper: def extract(sequence): return ("spam" if len(sequence) < 5 else sequence[5], sequence[2], "eggs" if len(sequence) < 5 else sequence[3], ) Raymond and Serhey are right: beyond a certain point, we should just write a custom function (whether lambda or not). We just disagree on which side of that point the single-default value case is, but I think we will agree that your example is so far past the point that we cannot even see it from here :-) -- Steve
Hi Vincent, Your idea is interesting but we are worried that there are not enough real use cases where it would be useful. Have you encountered situations yourself where this would make a difference? I am asking not for clarifying examples (you already provided one and from that it's perfectly clear to me what you are proposing) but for real-world code that would benefit from this addition to the itemgetter API. --Guido On Wed, May 2, 2018 at 1:08 AM, Vincent Maillol <vincent.maillol@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On Wed, May 02, 2018 at 10:08:55AM +0200, Vincent Maillol wrote:
I'm sorry, I'm not sure I understand what you mean by a global default. My interpretation of that would be the ability to set a global variable which acts as default for *all* calls to itemgetter. itemdefault = "spam" f = itemgetter(2, 99) f('abcd') # => returns ('c', 'spam') If that's what you mean, I hate it :-) [...]
I wouldn't call that a *global* default. If the default has to be specified on each call, I think that's both acceptible and desirable. Bug 14384 (https://bugs.python.org/issue14384) suggests that it might be more desirable to specify the default when you call the getter function, not when you call the factory: # this f = itemgetter(2, 99) f('abcd', default='spam') # => returns ('c', 'spam') # rather than this f = itemgetter(2, 99, default='spam') f('abcd') # => returns ('c', 'spam') I could go either way. In fact, at the risk of making a more complicated API, I'm almost inclined to allow both forms... -- Steve
On Wed, May 02, 2018 at 10:28:33PM -0700, Raymond Hettinger wrote:
Intended by whom? I think you are being too dismissive of actual use-cases requested by actual users. Default values might not have been the primary use considered when the API was first invented, but the fact that people keep asking for this feature should tell us that at least some people have intended uses that are remaining unmet.
At some point, we're really better off just using a lambda.
Maybe I'm slow today, but I'm having trouble seeing how to write this as a lambda. The single index case would be simple using the proposed binding- expression from PEP 572: # replace INDEX and DEFAULT with the values you want lambda seq: s[0] if (s := seq[INDEX:INDEX+1]) else DEFAULT but without the binding-expression, I have no idea how to do it cleanly and correctly in a lambda. For what it is worth, on my computer, itemgetter with a single index is 20% faster than a lambda using seq[INDEX] (where INDEX is hard-coded in the lambda), and 60% faster than a lambda using seq[INDEX:INDEX+1][0]. According to the bug report linked to earlier, one of the intended uses for itemgetter is when performance matters. -- Steve
On May 2, 2018, at 11:32 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Intended by whom?
By me. I proposed itemgetter() in the first place. That rationale I gave convinced Guido and python-dev to accept it. I then wrote the code, docs, tests and have maintained it for over a decade. So, I have a pretty good idea of what it was intended for.
I think you are being too dismissive of actual use-cases requested by actual users.
Wow, I don't know what to do with this. Over the years, I've added a lot of things requested by users. I really don't like the tone you've struck and what you've implied about me as developer. That feels somewhat pushy and aggressive. Why not just give a +1 to things that are a good idea and -1 for things we're better off without -- no need for ad hominem comments about the person making the post rather than its content -- that feels somewhat disrespectful.
When I've seen the request in the past, it always alway "it might be nice if ..." but there were no legitimate use cases presented, just toy examples. Also, I'm concerned that about increasing the complexity of itemgetter() API to serve an occasional exotic use case rather that being easy to learn and remember for the common cases. Raymond
On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote:
Maybe I'm slow today, but I'm having trouble seeing how to write this as a lambda.
Yes, I was definitely having a "cannot brain, I have the dumb" day, because it is not that hard to write using lambda. See discussion here: https://mail.python.org/pipermail/python-list/2018-May/732795.html If anything, the problem is a plethora of choices, where it isn't clear which if any is the best way, or the One Obvious Way. -- Steve
At one time, lambda was the one obvious way. Later, partial, itemgetter, attrgetter, and methodcaller were added to express common patterns for key-functions and map(). If needed, the zoo of lambda alternatives could be further extended to add a rpartial() function that partials from the right. That would have helped with Miki's example. Instead of: get = attrgetter('foo', None) return get(args) or get(config) or get(env) He could've written: get = rpartial(getattr, 'foo', None) return get(args) or get(config) or get(env) If itemgetter and attrgetter only did a single lookup, a default might make sense. However, that doesn't fit well with multiple and/or chained lookups where are number of options are possible. (See https://bugs.python.org/issue14384#msg316222 for examples and alternatives). Raymond
Hi everybody,
Have you encountered situations yourself where this would make a difference ? I need to get multiple objects in several dicts using the same set of keys.
I wanted to use itemgetter to reduce the number of lines of code but I have mixed up getattr/dict.get that have default parameter with attrgetter/itemgetter.
At some point, we're really better off just using a lambda. I kept severals line with `.get('key', value)`, I find that is more readable.
I did it just to try it with itemgetter https://github.com/Maillol/cpython/compare/master...Add-default-parameter-to... I don't know if we add parameter `default` to itemgetter, getitem, attrgetter the API will become more complexe or more consistency, but I agree with you, it is an occasional use case and we can always use `my_dict.get`. 2018-05-07 6:07 GMT+02:00 Raymond Hettinger <raymond.hettinger@gmail.com>:
(1) On 7 May 2018 at 01:07, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
That's somewhat hybrid, it would behave like: lambda obj: getattr(obj, "foo", None), but one might expect a "reversed partial" to have its arguments reversed as well, like: lambda obj: getattr(obj, None, "foo"). (2) Why getattr can't accept keyword arguments?
Since partial/partialmethod can apply keyword arguments, allowing the "default=" keyword argument in getattr would be an alternative to the "rpartial". (3) At one time, lambda was the one obvious way. [...]
I really like lambdas, but there are some "closure gotchas" like:
for idx in indices: ... do_something(lambda seq: seq[idx])
If the lambda is stored somewhere to be called after the loop ends (or after its iteration), the seq[idx] would load the item with the last iterated index. This:
for idx in indices: ... do_something(itemgetter(idx))
would behave more like this instead:
for idx in indices: ... do_something((lambda i: (lambda seq: seq[i]))(idx))
Which I wouldn't call "the one obvious way". (4) If itemgetter and attrgetter only did a single lookup, a default might make
As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default. Perhaps something like:
-- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)
As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default.
+1 Perhaps something like:
The keywords suffixed by indice are uncommon. Maybe we can use dedicated object.
Another possibility is that itemgetter becomes an object with `add_default` method.
or simply default parameter should be a list
itemgetter(-1, 0, 2, ... default=["first", itemgetter.NoDefault, "third"])
2018-05-08 1:37 GMT+02:00 Danilo J. S. Bellini <danilo.bellini@gmail.com>:
On Tue, May 08, 2018 at 10:13:43AM +0200, Vincent Maillol wrote:
Sorry Vincent, but I think that's precisely the sort of overly complicated API that Raymond was worried about when he voted against this proposal. A single default value that applies when the item (or key) isn't present is easy to comprehend. But this extention is, I think, a case of an over-engineered, excessively intricate solution, and I doubt it solves any real problem. And I'm not even sure I know what it means. You have keyword parameters default0, default1, and then *default* with no suffix. Is that a typo for default2, or is it supposed to be a default default, the default to use when no default is specified?
I'm afraid there's no elegance to this design either. I've had many occasions where I want to get an item, falling back on a default if it is not present. With dicts this is common enough that we have dict.get(). It is less convenient with lists, and I'd like to see itemgetter support that case. But I cannot think of any case where I have needed or wanted to extract item 5 with "spam" as the default item 2 with no default item 3 with "eggs" as the default (for example), let alone that this is *so common* that I'd rather read the complicated documenation to work out how to use the function, rather than just write a short helper: def extract(sequence): return ("spam" if len(sequence) < 5 else sequence[5], sequence[2], "eggs" if len(sequence) < 5 else sequence[3], ) Raymond and Serhey are right: beyond a certain point, we should just write a custom function (whether lambda or not). We just disagree on which side of that point the single-default value case is, but I think we will agree that your example is so far past the point that we cannot even see it from here :-) -- Steve
Hi Vincent, Your idea is interesting but we are worried that there are not enough real use cases where it would be useful. Have you encountered situations yourself where this would make a difference? I am asking not for clarifying examples (you already provided one and from that it's perfectly clear to me what you are proposing) but for real-world code that would benefit from this addition to the itemgetter API. --Guido On Wed, May 2, 2018 at 1:08 AM, Vincent Maillol <vincent.maillol@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
participants (7)
-
Danilo J. S. Bellini -
Guido van Rossum -
Raymond Hettinger -
Rob Cliffe -
Serhiy Storchaka -
Steven D'Aprano -
Vincent Maillol