Syntax for allowing extra keys when unpacking a dict as keyword arguments
Currently, unpacking a dict in order to pass its items as keyword arguments to a function will fail if there are keys present in the dict that are invalid keyword arguments: >>> def func(*, a): ... pass ... >>> func(**{'a': 1, 'b': 2}) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func() got an unexpected keyword argument 'b' The standard approach I have encountered in this scenario is to pass in the keyword arguments explicitly like so func( a=kwargs_dict["a"], b=kwargs_dict["b"], c=kwargs_dict["c"], ) But this grows more cumbersome as the number of keyword arguments grows. There are a number of other workarounds, such as using a dict comprehension to select only the required keys, but I think it would be more convenient to have this be a feature of the language. I don't know what a nice syntax for this would be, or even how feasible it is.
I could see this being an option, but to someone unfamiliar with it, it might seem strange that * unpacks iterables, ** unpacks dicts, and *** is a special thing only for keyword arguments that mostly behaves like **. On Friday, April 12, 2019 at 11:26:37 AM UTC-4, Bruce Leban wrote:
Bruce Leban wrote:
Unpacking json objects isn't something one does every day, so I don't think it would be worth adding syntax just for this. Rather than new syntax, how about a helper function such as: def call_with_kwds_from(func, kwds): code = func.__code__ names = code.co_varnames[:code.co_argcount] func(**{name : kwds[name] for name in names}) Usage example: def f(a, b): print("a =", a, "b =", b) d = {"a": "a_value", "b": "b_value", "c": "something extra"} call_with_kwds_from(f, d) -- Greg
I'm not a fan of this idea for two (related) reasons: 1) This seems like something that's a relatively niche case to *want* to do rather than doing unintentionally. 2) This vastly increases the chance of silent bugs in code. With regards to #1, the main examples I've seen posited in this thread are unpacking of unstructured object formats (e.g. JSON) and applying subsets of key-values from a configuration file [which is basically another unstructured object format). This seems like the level of functionality that I'd expect a helper function to be the right choice for - it can be part of the overall library handling the objects, or coded as a one-off for smaller projects. Regarding #2, passing unknown bags of values to functions is an easy way to run into both logic and security errors - see for example the problems that the Ruby on Rails ORM ran into with its batch-setter functionality when developers didn't properly limit what values in objects could be updated and forgot to sanitize inputs. Similarly, for keyword args with default values, adding this functionality would make it easy for a typo to result in a parameter that a user expects to be getting passed in not actually getting passed, because the function got {foobra=1} instead of {foobar=1} and silently discarded foobra=1 in favor of the default value for foobar. There may be cases where passing an unknown bag is what you want to do, but it's probably worth requiring that to be more explicit and intentional rather than an easily-missed extra asterisk. On Fri, Apr 12, 2019 at 4:09 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 4/12/2019 11:29 AM, Rhodri James wrote:
I don't want to speak for the OP, but I have a similar use case (which is why I wrote calllib). My use case is: I have number of callables that I don't control. I also have a dict of parameters that the callables might take as parameters. I want to call one of the callables, passing only the subset of parameters that that particular callable takes. Eric
On 12 Apr 2019, at 19:16, Eric V. Smith <eric@trueblade.com> wrote:
I don't want to speak for the OP, but I have a similar use case (which is why I wrote calllib). My use case is: I have number of callables that I don't control. I also have a dict of parameters that the callables might take as parameters. I want to call one of the callables, passing only the subset of parameters that that particular callable takes.
Could you expand on "that I don't control"? Where do these come from? We have similar used cases in libs we've created but there we define that the API is that you must do **_ to be compatible with future versions of the lib. / Anders
On 4/12/2019 4:00 PM, Anders Hovmöller wrote:
Their names come from a config file. I dynamically load them and call them. I wrote some of them, other people wrote others. It's an internal corporate app, and there are a lot of callables, all with different release schedules and controlled by different teams. Over time, the interface to these callables has expanded. First, it took just x, then a few of them needed x and y, and others needed x and z. I realize this isn't the greatest interface, and in an ideal world we would have come up with a better way to specify this. But it evolved over time, and it is what it is.
We have similar used cases in libs we've created but there we define that the API is that you must do **_ to be compatible with future versions of the lib.
Unfortunately I don't control the interface or the source to the code I'm calling, so the best I've been able to do is only call each function with the parameters I know it expects, based on its signature. At least the parameter names are well defined. Eric
Re-ordered to avoid top-posting... On 12/04/2019 18:50, Viktor Roytman wrote:
Hang on, I missed this first time around. This gives you exactly the same problem: Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def func(*, a): ... pass ... >>> d = {"a":1, "b":2} >>> func(**d) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func() got an unexpected keyword argument 'b' >>> func(a=d["a"], b=d["b"]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func() got an unexpected keyword argument 'b' >>>
That's not the same issue at all, if I'm understanding you correctly. In any case, surely you need to do some validation on your dictionary of keyword arguments? Otherwise you are setting yourself up for a world of pain. If you do that, you should take the opportunity to decide what to do with invalid keys. -- Rhodri James *-* Kynesim Ltd
The specific pain point that motivated this was constructing many interrelated models using a dict. So, for example, if there is a User model with a related Address model, and the input is user_kwargs = dict( name='user', age=20, address=dict( city='city', state='ST', ), ) then passing this information in to the User constructor directly will fail, because User.address is a related model, not a simple field. I agree that most of the time you should want unexpected keyword arguments to raise an exception, but in specific circumstances it can be helpful to extract only the pieces of a dict that are relevant.
On Sat, Apr 13, 2019 at 1:12 AM Viktor Roytman <viktor.roytman@gmail.com> wrote:
I'm not 100% sure I understand your proposal, so I'm going to restate it; anywhere that I'm misrepresenting you, please clarify! Given this function and this dictionary: def func(*, a): pass args = {'a': 1, 'b': 2} you want to call the function, passing the recognized argument 'a' the value from the dict, but ignoring the superfluous 'b'. Are you able to alter the function? If so, just add kwargs to it: def func(*, a, **_): pass and then any unrecognized args will quietly land in the junk dictionary. ChrisA
Le 12/4/19 à 18:02, Viktor Roytman a écrit :
This seems like a very specific problem. And in that case, I'm not sure that the solution should come from the language, but there definitely are ways to do it using an intermediary calling function. That calling function would just do one of the following things: * inspect the signature of the called function and only pass the arguments it needs * contain a lookup table for which arguments are required by which function and use it to only pass the ones it needs * or probably what I would do: def call_func(func_name, **kwargs): try: return globals()[func_name](**kwargs) # Not sure how you're doing it except TypeError as e: stripped_arg = e[len(func_name) + 39:-1] # There's probably a more elegant way of doing it # Maybe warn the user that he's using an API that is under depreciation process, so that the authors may know they should add **kwargs to its signature kwargs = del kwargs[stripped_arg] return call_func(func_name, **kwargs) # Try to make the call again (I haven't tried the thing, it's just an idea of what I'd probably do in your situation)
I don't think it's possible to make this work reliably. In particular, it's an important feature of python that you can make wrappers that pass through arguments and are equivalent to the original function: def original(a=0): ... def wrapper(*args, **kwargs): return original(*args, **kwargs) Right now these can be called in exactly the same ways. But with the proposal they would become different: # ok original(***{"a": 1, "b": 2}) # raises TypeError wrapper(***{"a": 1, "b": 2}) The problem is that the extra star gets lost when passing through the wrapper. In this case you might be able to fix this by using functools.wraps to fix up the signature introspection metadata, but that doesn't work in more complex cases (e.g. when the wrapper adds/removes some args while passing through the rest). In Python, signature introspection is a best-effort thing, and IME not super reliable. -n On Fri, Apr 12, 2019, 08:11 Viktor Roytman <viktor.roytman@gmail.com> wrote:
On Saturday, April 13, 2019 at 9:03:54 AM UTC-4, Chris Angelico wrote:
I like this idea. It accomplishes the goal of only using some of the keys and has broader applications for working with dicts in general. It might also be nice to have something that splits a dict into two, like >>> items = dict(a=1, b=2, c=3) >>> included, excluded = items &- {'a', 'b'} >>> print(included) {'a': 1, 'b': 2} >>> print(excluded) {'c': 3} I don't know if I like the &- "operator" but it is illustrative.
On Sat, Apr 13, 2019 at 9:02 AM Chris Angelico <rosuav@gmail.com> wrote:
Would be really nice to be able to spell this as a dict/set intersection.
func(**(d & {'a', 'b', 'c'}))
That would be _very_ consistent with the ongoing discussions about operators over dicts. -- Juancarlo *Añez*
I could see this being an option, but to someone unfamiliar with it, it might seem strange that * unpacks iterables, ** unpacks dicts, and *** is a special thing only for keyword arguments that mostly behaves like **. On Friday, April 12, 2019 at 11:26:37 AM UTC-4, Bruce Leban wrote:
Bruce Leban wrote:
Unpacking json objects isn't something one does every day, so I don't think it would be worth adding syntax just for this. Rather than new syntax, how about a helper function such as: def call_with_kwds_from(func, kwds): code = func.__code__ names = code.co_varnames[:code.co_argcount] func(**{name : kwds[name] for name in names}) Usage example: def f(a, b): print("a =", a, "b =", b) d = {"a": "a_value", "b": "b_value", "c": "something extra"} call_with_kwds_from(f, d) -- Greg
I'm not a fan of this idea for two (related) reasons: 1) This seems like something that's a relatively niche case to *want* to do rather than doing unintentionally. 2) This vastly increases the chance of silent bugs in code. With regards to #1, the main examples I've seen posited in this thread are unpacking of unstructured object formats (e.g. JSON) and applying subsets of key-values from a configuration file [which is basically another unstructured object format). This seems like the level of functionality that I'd expect a helper function to be the right choice for - it can be part of the overall library handling the objects, or coded as a one-off for smaller projects. Regarding #2, passing unknown bags of values to functions is an easy way to run into both logic and security errors - see for example the problems that the Ruby on Rails ORM ran into with its batch-setter functionality when developers didn't properly limit what values in objects could be updated and forgot to sanitize inputs. Similarly, for keyword args with default values, adding this functionality would make it easy for a typo to result in a parameter that a user expects to be getting passed in not actually getting passed, because the function got {foobra=1} instead of {foobar=1} and silently discarded foobra=1 in favor of the default value for foobar. There may be cases where passing an unknown bag is what you want to do, but it's probably worth requiring that to be more explicit and intentional rather than an easily-missed extra asterisk. On Fri, Apr 12, 2019 at 4:09 PM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 4/12/2019 11:29 AM, Rhodri James wrote:
I don't want to speak for the OP, but I have a similar use case (which is why I wrote calllib). My use case is: I have number of callables that I don't control. I also have a dict of parameters that the callables might take as parameters. I want to call one of the callables, passing only the subset of parameters that that particular callable takes. Eric
On 12 Apr 2019, at 19:16, Eric V. Smith <eric@trueblade.com> wrote:
I don't want to speak for the OP, but I have a similar use case (which is why I wrote calllib). My use case is: I have number of callables that I don't control. I also have a dict of parameters that the callables might take as parameters. I want to call one of the callables, passing only the subset of parameters that that particular callable takes.
Could you expand on "that I don't control"? Where do these come from? We have similar used cases in libs we've created but there we define that the API is that you must do **_ to be compatible with future versions of the lib. / Anders
On 4/12/2019 4:00 PM, Anders Hovmöller wrote:
Their names come from a config file. I dynamically load them and call them. I wrote some of them, other people wrote others. It's an internal corporate app, and there are a lot of callables, all with different release schedules and controlled by different teams. Over time, the interface to these callables has expanded. First, it took just x, then a few of them needed x and y, and others needed x and z. I realize this isn't the greatest interface, and in an ideal world we would have come up with a better way to specify this. But it evolved over time, and it is what it is.
We have similar used cases in libs we've created but there we define that the API is that you must do **_ to be compatible with future versions of the lib.
Unfortunately I don't control the interface or the source to the code I'm calling, so the best I've been able to do is only call each function with the parameters I know it expects, based on its signature. At least the parameter names are well defined. Eric
Re-ordered to avoid top-posting... On 12/04/2019 18:50, Viktor Roytman wrote:
Hang on, I missed this first time around. This gives you exactly the same problem: Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def func(*, a): ... pass ... >>> d = {"a":1, "b":2} >>> func(**d) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func() got an unexpected keyword argument 'b' >>> func(a=d["a"], b=d["b"]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func() got an unexpected keyword argument 'b' >>>
That's not the same issue at all, if I'm understanding you correctly. In any case, surely you need to do some validation on your dictionary of keyword arguments? Otherwise you are setting yourself up for a world of pain. If you do that, you should take the opportunity to decide what to do with invalid keys. -- Rhodri James *-* Kynesim Ltd
The specific pain point that motivated this was constructing many interrelated models using a dict. So, for example, if there is a User model with a related Address model, and the input is user_kwargs = dict( name='user', age=20, address=dict( city='city', state='ST', ), ) then passing this information in to the User constructor directly will fail, because User.address is a related model, not a simple field. I agree that most of the time you should want unexpected keyword arguments to raise an exception, but in specific circumstances it can be helpful to extract only the pieces of a dict that are relevant.
On Sat, Apr 13, 2019 at 1:12 AM Viktor Roytman <viktor.roytman@gmail.com> wrote:
I'm not 100% sure I understand your proposal, so I'm going to restate it; anywhere that I'm misrepresenting you, please clarify! Given this function and this dictionary: def func(*, a): pass args = {'a': 1, 'b': 2} you want to call the function, passing the recognized argument 'a' the value from the dict, but ignoring the superfluous 'b'. Are you able to alter the function? If so, just add kwargs to it: def func(*, a, **_): pass and then any unrecognized args will quietly land in the junk dictionary. ChrisA
Le 12/4/19 à 18:02, Viktor Roytman a écrit :
This seems like a very specific problem. And in that case, I'm not sure that the solution should come from the language, but there definitely are ways to do it using an intermediary calling function. That calling function would just do one of the following things: * inspect the signature of the called function and only pass the arguments it needs * contain a lookup table for which arguments are required by which function and use it to only pass the ones it needs * or probably what I would do: def call_func(func_name, **kwargs): try: return globals()[func_name](**kwargs) # Not sure how you're doing it except TypeError as e: stripped_arg = e[len(func_name) + 39:-1] # There's probably a more elegant way of doing it # Maybe warn the user that he's using an API that is under depreciation process, so that the authors may know they should add **kwargs to its signature kwargs = del kwargs[stripped_arg] return call_func(func_name, **kwargs) # Try to make the call again (I haven't tried the thing, it's just an idea of what I'd probably do in your situation)
I don't think it's possible to make this work reliably. In particular, it's an important feature of python that you can make wrappers that pass through arguments and are equivalent to the original function: def original(a=0): ... def wrapper(*args, **kwargs): return original(*args, **kwargs) Right now these can be called in exactly the same ways. But with the proposal they would become different: # ok original(***{"a": 1, "b": 2}) # raises TypeError wrapper(***{"a": 1, "b": 2}) The problem is that the extra star gets lost when passing through the wrapper. In this case you might be able to fix this by using functools.wraps to fix up the signature introspection metadata, but that doesn't work in more complex cases (e.g. when the wrapper adds/removes some args while passing through the rest). In Python, signature introspection is a best-effort thing, and IME not super reliable. -n On Fri, Apr 12, 2019, 08:11 Viktor Roytman <viktor.roytman@gmail.com> wrote:
On Saturday, April 13, 2019 at 9:03:54 AM UTC-4, Chris Angelico wrote:
I like this idea. It accomplishes the goal of only using some of the keys and has broader applications for working with dicts in general. It might also be nice to have something that splits a dict into two, like >>> items = dict(a=1, b=2, c=3) >>> included, excluded = items &- {'a', 'b'} >>> print(included) {'a': 1, 'b': 2} >>> print(excluded) {'c': 3} I don't know if I like the &- "operator" but it is illustrative.
On Sat, Apr 13, 2019 at 9:02 AM Chris Angelico <rosuav@gmail.com> wrote:
Would be really nice to be able to spell this as a dict/set intersection.
func(**(d & {'a', 'b', 'c'}))
That would be _very_ consistent with the ongoing discussions about operators over dicts. -- Juancarlo *Añez*
participants (11)
-
Amber Yust -
Anders Hovmöller -
Brice Parent -
Bruce Leban -
Chris Angelico -
Eric V. Smith -
Greg Ewing -
Juancarlo Añez -
Nathaniel Smith -
Rhodri James -
Viktor Roytman