[Python-ideas] kwargs for return

[Steven D'Aprano]

On Sun, Jan 27, 2019 at 03:33:15PM +1100, Cameron Simpson wrote:

> I don't think so. It looks to me like Thomas' idea is to offer a 
> facility a little like **kw in function, but for assignment.

Why **keyword** arguments rather than **positional** arguments?

Aside from the subject line, what part of Thomas' post hints at the 
analogy with keyword arguments?

    function(spam=1, eggs=2, cheese=3)

Aside from the subject line, I'm not seeing the analogy with keyword 
parameters here. If he wants some sort of dict unpacking, I don't think 
he's said so. Did I miss something?

But in any case, regardless of whether he wants dict unpacking or 
not, Thomas doesn't want the caller to be forced to update their calls. 
Okay, let's consider the analogy carefully:

Functions that collect extra keyword args need to explicitly include a 
**kwargs in their parameter list. If we write this:

    def spam(x):
        ...

    spam(123, foo=1, bar=2, baz=3)

we get a TypeError. We don't get foo, bar, baz silently ignored. So if 
we follow this analogy, then dict unpacking needs some sort of "collect 
all remaining keyword arguments", analogous to what we can already do 
with sequences:

    foo, bar, baz, *extras = [1, 2, 3, 4, 5, 6, 7, 8]

Javascript ignores extra values:

js> var [x, y] = [1, 2, 3, 4]
js> x
1
js> y
2

but in Python, this is an error:

    foo, bar, baz = [1, 2, 3, 4]


So given some sort of "return a mapping of keys to values":

   def spam():
       # For now, assume we simply return a dict
       return dict(messages=[], success=True)

let's gloss over the dict-unpacking syntax, whatever it is, and assume 
that if a function returns a *single* key:value, and the assignment 
target matches that key, it Just Works:

    success = spam()

But by analogy with **kwargs that has to be an error since there is 
nothing to collect the unused key 'messages'. It needs to be:

    success, **extras = spam()

which gives us success=True and extras={'messages': []}.

But Thomas doesn't want the caller to have to update their code either. 
To do so would be analogous to having function calls start ignoring 
unexpected keyword arguments:

    assert len([], foo=1, bar=2) == 0

so *not* like **kwargs at all. And it would require ignoring the Zen:

Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.


> So in his case, he wants to have one backend start returning a richer 
> result _without_ bringing all the other backends up to that level. This 
> is particularly salient when "the other backends" includes third party 
> plugin facilities, where Thomas (or you or I) cannot update their 
> source.

I've pointed out that we can solve his use-case by planning ahead and 
returning an object that can hold additional, optional fields. Callers 
that don't know about those fields can just ignore them. Backends that 
don't know to supply optional fields can just leave them out. 

Or he can wrap the backend functions in one of at least three Design 
Patterns made for this sort of scenario: Adaptor, Bridge or Facade, 
whichever is more appropriate. By decoupling the backend from the 
frontend, he can easily adapt the result even if he cannot change the 
backends directly.

So Thomas' use-case already has good solutions. But as people have 
repeatedly pointed out, they all require some foresight and planning. 

To which my response is, yes they do. Just like we have to plan ahead 
and include *extra in your sequence packing assignments, or **kwargs in 
your function parameter list.


> So, he wants to converse of changing a function which previously was 
> like:
> 
>  def f(a, b):
> 
> into:
> 
>  def f(a, b, **kw):

Yes, but to take this analogy further, he wants to do so without having 
to actually add that **kw to the parameter list. So he apparently wants 
errors to pass silently.

Since Thomas apparently feels that neither the caller nor the callee 
should be expected to plan ahead, while still expecting backwards 
compatibility to hold even in the event of backwards incompatible 
changes, I can only conclude that he wants the interpreter to guess the 
intention of the caller AND the callee and Do The Right Thing no matter 
what:

http://www.catb.org/jargon/html/D/DWIM.html

(Half tongue in cheek here.)


> In Python you can freely do this without changing _any_ of the places 
> calling your function.

But only because the function author has included **kw in their 
parameter list. If they haven't, it remains an error.


> So, for assignment he's got:
> 
>  result = backend.foo()
> 
> and he's like to go to something like:
> 
>  result, **kw = richer_backend.foo()
> 
> while still letting the older less rich backends be used in the same 
> assignment.

That would be equivalent to having unused keyword arguments (or 
positional arguments for that matter) just disappear into the aether, 
silently with no error or notice. Like in Javascript.

And what about the opposite situation, where the caller is expecting two 
results, but the backend only returns one? Javascript packs the extra 
variable with ``undefined``, but Python doesn't do that.

Does Thomas actually want errors to pass silently? I don't wish to 
guess his intentions.


[...]
> Idea: what if **kw mean to unpack RHS.__dict__ (for ordinary objects) 
> i.e. to be filled in with the attributes of the RHS expression value.
> 
> So, Thomas' old API:
> 
>    def foo():
>      return 3
> 
> and:
> 
>  a, **kw = foo()
> 
> get a=3 and kw={}.

Um, no, it wouldn't do that -- it would fail, because ints don't have a 
__dict__. And don't forget __slots__. What about properties and other 
descriptors, private attributes, etc.

Is *every* attribute of an object supposed to be a separate part of the 
return result?

If a caller knows about the new API, how to they directly access the 
newer fields? You might say:

    a, x, y, **kwargs = foo()

to automatically extract a.x and a.y (as in the example class you gave 
below) but what if I want to give names which are meaningful at the 
caller end, instead of using the names foo() supplies?

    a, counter, description, **kwargs = foo()

Now my meaningful names don't match the attributes. Nor does the 
order I give them. Now what happens?


> But the richer API:
> 
>  class Richness(int):
> 
>    def __init__(self, value):
>      super().__int__(value)
>      self.x = 'x!'
>      self.y = 4

[...]
> I've got mixed mfeelings about this

I don't.



-- 
Steve