[Python-ideas] Preserving **kwargs order

[Eric Snow]

On Thu, Apr 3, 2014 at 7:50 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> First, not all wrappers are usually used as decorators. The most obvious example is partial, which is why I raised it before. I've also applied lru_cache, or a TTL-based cache that I got from some module on PyPI, to lookup functions that I pulled out of another module. And so on. Any such general-purpose wrappers that work today would need to be replaced after your change. It's not at all localized in my code; it's not even all in my code. The guy who wrote TTLCache has no idea whether I want to use it for keyword-order-preserving functions. (Well, today, he knows that I _don't_ want to do so, because it's not possible...)

So you are saying that it is not uncommon to use "perfect passthrough"
decorators/wrappers you do not control.  In my experience it is rare
outside the stdlib, but then again I gladly admit that my experience
has not involved software domains where apparently such takes place.
Regardless, the workarounds are the same.  Likewise they add the same
complication.  Perhaps it does favor the decorator syntax over the
__kworder__ local variable.

> Well, that has nothing to do with the point I raised, but actually, it raises another (smaller) problem. The fact that a function has been wrapped in a decorator doesn't change its signature, as in the thing you can inspect with things like functools.getfullargspec or functools.signature.
>
> So, just adding this decorator, or even adding an inspect function that looks for whatever the decorator does (like setting a new bit on co_flags), isn't sufficient; you need to work out how it should affect inspect.Signature and/or inspect.Parameter objects. (Maybe there's just a new kind, VAR_ORDERED_KEYWORD... but the fact that Parameters of this kind would have to have the same str() as VAR_KEYWORD means most people still wouldn't see it...)

Looking into the implications on inspect.signature, et al. is on my
to-do list. :)

> Well, you said before that the perfect-forwarding wrapper problem could be solved easily. I asked you to explain how. Now it seems like your answer is that it can't be solved, short of by rewriting every wrapper function that might ever be used on order-preserving functions. Which is exactly what I said, and you denied, in the first place.

I stand by what I said.  I don't think the impact is that severe and I
think that solving the issue wouldn't be that hard and I do agree that
it adds complication that would be worth avoiding.  I think your
summary exaggerates things a bit.

You are correct that I did not give any concrete solution to resolve
the perfect-forwarding wrapper case, even in the face of your concrete
example.  If, when we have a better idea of the various options, one
of the original two (or any other opt-in solution) stands above the
rest I'll be glad to spend more time looking on how to make the
pass-through case work with less complication.

>
>>> But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders.
>>
>>Nice.
>
>
> The decorator may be nice, but the fact that you have to add that decorator to ever wrapper function defined in every general-purpose decorator or other wrapper-generator in the world does not seem nice to me.

It only matters for the ones you are applying to your functions that
need to preserve keyword order.  I agree that any fix that you can't
easily apply to a function you do not "own" isn't very helpful.

>
>>That said, I agree that it would be best to come up with a solution that drops in without breaking the "perfect forwarders", thus rendering those tools unnecessary.  Here are 3 ideas I had last night and today:
>
>>
>>
>>1. Add a '__order__' attribute to dict that defaults to None, and always set it for kwargs (to a frozen list/tuple).
>
> A dict with an __order__ attrib is just a less-useful but nearly-as-heavy-weight equivalent to an OrderedDict. What about the cost of always using an OrderedDict makes it unacceptable? How much better is this?

Guido's concern was that in some uncommon cases the difference in
performance (both memory and speed) would render OrderedDict
undesirable for  **kwargs.  By storing the initial order on a dict
attribute the only overhead would be 1 pointer per dict plus the size
of the container.

>
> And it doesn't actually work.
>
> First, the **d calling syntax unpacks the keywords in d's iteration order. That works fine for OrderedDict, but not for a dict that has an order but ignores it while iterating. So you need an additional rule (and additional code in each implementation) to make the **d calling syntax use __order__ order instead of iteration order (and deal with all of the edge cases, like when __order__ is missing some keys, or has keys the dict doesn't, etc.).
>
> And, even after that, any wrapper that modifies its kwargs instead of passing them through unchanged (like, again, partial) could easily break __order__ unless you also change all such wrappers to handle it properly. Again, not a problem for OrderedDict, because OrderedDict automatically handles it properly.

Correct, the interpreter would have to take __order__ into account
when unpacking kwargs into a function call.  I realized earlier that I
hadn't mentioned that.

>
>>2. Use a minimal dict subclass just for kwargs that provides __order__ (meaning all other uses of dict don't take the memory hit).
>
>
> That still has the other problems as #1.
>
>>3. Use OrderedDict for kwargs by default, and provide a decorator that people can use to get just a dict (a.k.a. the current behavior).  It's only a few extreme cases where OrderedDict is problematic.
>
>
> So how do C functions (which I'm guessing are many of those extreme cases...) get just a dict?

Good question.  I already plan on addressing the impact of the
proposal on the C-API as part of the PEP.

>
> Also, is it actually true that Guido rejected the idea of "kwargs is always an OrderedDict" out of hand because a few extreme cases are problematic, or is that just a guess?
>

He indicated that any solution would have to demonstrate that it did
not have any significant impact on performance in those edge cases.
[1][2][3]

-eric


[1] https://mail.python.org/pipermail/python-dev/2013-May/126328.html
[2] https://mail.python.org/pipermail/python-ideas/2013-February/019699.html
[3] https://mail.python.org/pipermail/python-ideas/2013-February/019704.html