[Python-ideas] Preserving **kwargs order

[Andrew Barnert]

From: Eric Snow <ericsnowcurrently at gmail.com>
Sent: Thursday, April 3, 2014 4:37 PM


>On Apr 3, 2014 2:55 AM, "Andrew Barnert" <abarnert at yahoo.com> wrote:
>> It's definitely a big deal. In Python 3.4, it is trivial to write a wrapper that perfectly forwards its arguments. Or that modifies its arguments in somewhat, then perfectly forwards the modified version. This is critical to being able to write decorators. Nearly every existing decorator in the stdlib, third-party libs, ActiveState recipes, and even your own projects does this. I'll show a dead-simple example below.
>>
>> But with either of your options, that would no longer be true. It would be at least a little more difficult to write a wrapper that perfectly forwards its arguments, and every existing decorator in the world would have to be rewritten to do so.
>
>I'll concede "a little more difficult" in the pass-through case (with unaware decorators being the main issue).  :)  One reason why I don't think it's a huge problem in the decorator case is that you decorate your own functions (though not necessarily with your own decorators).  So if you change an existing function to use ordered kwargs, then you would already be focused on the feature.  You would ensure any decorators you've applied to the function accommodate ordered pass-through, perhaps adapting/wrapping the decorators to do so.  It's all pretty localized in your code, so it should be hard to miss.

First, not all wrappers are usually used as decorators. The most obvious example is partial, which is why I raised it before. I've also applied lru_cache, or a TTL-based cache that I got from some module on PyPI, to lookup functions that I pulled out of another module. And so on. Any such general-purpose wrappers that work today would need to be replaced after your change. It's not at all localized in my code; it's not even all in my code. The guy who wrote TTLCache has no idea whether I want to use it for keyword-order-preserving functions. (Well, today, he knows that I _don't_ want to do so, because it's not possible…) 

>Other than the pass-through case, the 2 outstanding approaches seem fine.  Existing functions would probably need a signature change anyway and for new functions no one has to change any code. :)  Either way, if a function depends on ordered kwargs, then that's the sort of important feature that you will make clear to people using your function, on par with the rest of the signature.


Well, that has nothing to do with the point I raised, but actually, it raises another (smaller) problem. The fact that a function has been wrapped in a decorator doesn't change its signature, as in the thing you can inspect with things like functools.getfullargspec or functools.signature.

So, just adding this decorator, or even adding an inspect function that looks for whatever the decorator does (like setting a new bit on co_flags), isn't sufficient; you need to work out how it should affect inspect.Signature and/or inspect.Parameter objects. (Maybe there's just a new kind, VAR_ORDERED_KEYWORD… but the fact that Parameters of this kind would have to have the same str() as VAR_KEYWORD means most people still wouldn't see it…)

>> Can Python fix this for you magically?

>
>I'll drop this proposal in a heartbeat if it requires something like that!  I just don't think it does, even with the 2 less-optimal solutions we've been discussing.  (My definition of magic may be different than yours though. <wink>)


Well, you said before that the perfect-forwarding wrapper problem could be solved easily. I asked you to explain how. Now it seems like your answer is that it can't be solved, short of by rewriting every wrapper function that might ever be used on order-preserving functions. Which is exactly what I said, and you denied, in the first place.

>> But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders.
>
>Nice.


The decorator may be nice, but the fact that you have to add that decorator to ever wrapper function defined in every general-purpose decorator or other wrapper-generator in the world does not seem nice to me.

>That said, I agree that it would be best to come up with a solution that drops in without breaking the "perfect forwarders", thus rendering those tools unnecessary.  Here are 3 ideas I had last night and today:

>
>
>1. Add a '__order__' attribute to dict that defaults to None, and always set it for kwargs (to a frozen list/tuple).

A dict with an __order__ attrib is just a less-useful but nearly-as-heavy-weight equivalent to an OrderedDict. What about the cost of always using an OrderedDict makes it unacceptable? How much better is this?

And it doesn't actually work.

First, the **d calling syntax unpacks the keywords in d's iteration order. That works fine for OrderedDict, but not for a dict that has an order but ignores it while iterating. So you need an additional rule (and additional code in each implementation) to make the **d calling syntax use __order__ order instead of iteration order (and deal with all of the edge cases, like when __order__ is missing some keys, or has keys the dict doesn't, etc.).

And, even after that, any wrapper that modifies its kwargs instead of passing them through unchanged (like, again, partial) could easily break __order__ unless you also change all such wrappers to handle it properly. Again, not a problem for OrderedDict, because OrderedDict automatically handles it properly.

>2. Use a minimal dict subclass just for kwargs that provides __order__ (meaning all other uses of dict don't take the memory hit).


That still has the other problems as #1.

>3. Use OrderedDict for kwargs by default, and provide a decorator that people can use to get just a dict (a.k.a. the current behavior).  It's only a few extreme cases where OrderedDict is problematic.


So how do C functions (which I'm guessing are many of those extreme cases…) get just a dict?

Also, is it actually true that Guido rejected the idea of "kwargs is always an OrderedDict" out of hand because a few extreme cases are problematic, or is that just a guess?