[Python-ideas] Preserving **kwargs order

[Andrew Barnert]

On Apr 2, 2014, at 23:39, Eric Snow <ericsnowcurrently at gmail.com> wrote:


> On Thu, Mar 20, 2014 at 10:28 AM, Andrew Barnert <abarnert at yahoo.com> wrote:
>> And as far as I can see, there is no way to solve this problem. The only way to add keyword order information without breaking all existing forwarding functions is to make the **kwargs parameter always preserve order (and make the **kwargs calling syntax preserve order if present)--that is, to make it always an OrderedDict, which Guido has already rejected.
> 
> It's not all that bleak.  Either of the options I outlined should
> still work, though option #1 (using a decorator) is better for the
> pass-through case that has come up (which I'm not convinced is such a
> big deal).
It's definitely a big deal. In Python 3.4, it is trivial to write a wrapper that perfectly forwards its arguments. Or that modifies its arguments in somewhat, then perfectly forwards the modified version. This is critical to being able to write decorators. Nearly every existing decorator in the stdlib, third-party libs, ActiveState recipes, and even your own projects does this. I'll show a dead-simple example below.


But with either of your options, that would no longer be true. It would be at least a little more difficult to write a wrapper that perfectly forwards its arguments, and every existing decorator in the world would have to be rewritten to do so.

Let's take your specific example, and simplify it even further. I'll show why it doesn't _obviously_ work, and then you try to explain how you could make it work anyway:

    @preserve_kwargs_order
    def spam(**kwargs):
        print(kwargs)

    spam(a=1, b=2)

So far, so good; your proposal means spam is guaranteed to get an OrderedDict with a before b.

Now let's take a simple and useful general-purpose wrapper:

    def memoize(func):
        cache = {}
        def wrapper(*args, **kwargs):
            if (args, kwargs) not in _cache:
                cache[args, kwargs] = func(*args, **kwargs)
            return cache[args, kwargs]
        return wrapper

    eggs = memoize(spam)
    eggs(a=1, b=2)

Here, spam is guaranteed to get an OrderedDict, but it's arbitrary whether a comes before or after b. Why? Well, inside eggs (aka wrapper), kwargs is a plain old dict. So when it calls spam (aka func) with **kwargs, they get sent in whatever arbitrary order they had in the dict.

Would anyone ever write or use a wrapper like this? Well, the Python developers thought it was useful enough to add a more complicated version of the same thing to the stdlib as functools.lru_cache, and there were dozens of recipes and third-party modules doing the same thing before it was added to the stdlib, so, yes. And, more importantly, if you work through any other general-purpose wrapper, it's going to have the exact same problems. (It's marginally harder to work through, but maybe more obvious once you do so, with wrappers that modify their arguments, like functools.partial.)

Can Python fix this for you magically? There are four potential places it could, and none of them will work. When compiling memoize, Python obviously has no idea whether the value (or values) you will later pass to it are order-preserving functions or normal functions. At runtime, when executing that compiled memoize definition, you still have the same problems. Later, when calling memoize on spam, Python now knows that the value is an order-preserving function—but it no longer knows that memoize is intended to be a perfect-forwarding wrapper-generator. (In fact, there is nothing special about the definition of memoize in the first place that makes it detectably a perfect-forwarding wrapper at any point. If that isn't obvious for memoize, consider functools.lru_cache or functools.partial.) And finally, when calling the memoized function eggs, it's obviously still too late.So, there is no place that Python could automatically figure out that it
 needs to push the order-preservation upward from spam to eggs.

The only way to make eggs end up as order-preserving is to change Python so that all functions (or at least all functions defined inside other functions) are order-preserving.

Either you or someone else suggested that if Python could just pass the kwargs dict straight through instead of unpacking and repacking it, that would help. But it wouldn't. There's nothing marking eggs as an order-preserving function, so its kwargs is just a plain dict, and passing that straight through can't help anything. And besides, that idea isn't possible, or even (in general) sensible. Consider eggs=partial(spam, c=3), where the kwargs in eggs only has a and b, but the one inside spam has a, b, and c; obviously they can't be the same mapping. Or consider your original example (with explicit a and b params on spam before **kwargs) with memoize, where the kwargs in eggs has a, b, c, and d, but the one inside spam has only c and d.

Now, obviously you could fix any particular wrapper. Presumably your magic decorator works by doing something detectable from Python, like a new co_flags flag. So, we could define two new functions (presumably in inspect and functools, respectively):


    def preserves_kwargs_order(func):
        return func.__code__.co_flags & 16

    def wrap_preserve_kwargs_order(func):
        def wrapper(inner):
            if inspect.preserves_kwargs_order(func):
                return preserve_kwargs_order(inner)
            else:
                return inner
        return wrapper

And now, all you have to do is add one line to most decorators and they're once again perfect forwarders:

    def memoize(func):

        _cache = {}
        @functools.wrap_preserve_kwargs_order(func)
        def wrapper(*args, **kwargs):
            if (args, kwargs) not in _cache:
                _cache[args, kwargs] = func(*args, **kwargs)
            return _cache[args, kwargs]
        return wrapper


But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders.

You could even add that wrap_preserve_kwargs_order call into functools.wraps, and that would fix _many_ decorators (because many decorators are written to use functools.wraps), but still not all. Most notably, the wrappers inside functools don't use wraps (even if you don't get the C-accelerated versions). So, that would still means that you have to add the new decorator to every wrapper function in the world that doesn't use wraps.