On Apr 3, 2014 11:42 PM, "Andrew Barnert" <abarnert@yahoo.com> wrote:
From: Eric Snow <ericsnowcurrently@gmail.com>
Sent: Thursday, April 3, 2014 8:03 PM
On Thu, Apr 3, 2014 at 7:50 PM, Andrew Barnert <abarnert@yahoo.com>
wrote:
[snip]
So you are saying that it is not uncommon to use "perfect passthrough" decorators/wrappers you do not control.
Yes. Just my first example, partial, is ridiculously common, and it's a
wrapper I don't control. Half the servers, GUIs, and other callback-based programs I've worked on need partial functions, and get them either by using partial, or by using lambda functions instead--which has an equivalent and parallel problem that, come to think of it, still requires a solution too...
However, what we _don't_ know is how common it would be to use such
wrappers with functions that need ordered kwargs. The prototypical case of using partial on GUI or server callbacks obviously doesn't apply there--in a Tkinter GUI, your callback functions almost never take kwargs, and, if they did, it would probably be just to pass the dict as a widget configure dict or something like that, which couldn't possibly care about the order. But really, _no_ example that works today could apply; it's not possible to write functions that care about their keyword order, so we can only try to guess whether the potential use cases for them could involve wrapping them or not.
Regardless, the workarounds are the same. Likewise they add the same complication. Perhaps it does favor the decorator syntax over the __kworder__ local variable.
I'm not sure it favors either. Either way, every function that returns a
forwarding wrapper will need to be changed to apply the workaround. The workaround itself may be easier, or at least prettier, with the decorator, but the hard part isn't the workaround itself (as I showed, it could just be a simple on-line change), but figuring out which wrappers need it, and possibly copying and forking them into your own code, so you can apply the workaround to each one. Right. That's why a ordered-by-default approach is favorable. Providing an opt-out mechanism should satisfy what I still think are uncommon cases.
[snip]
Looking into the implications on inspect.signature, et al. is on my to-do list. :)
After a little more thought, what's really needed is a consistent way to
represent ordered-kwargs parameters in *documentation*.
Once you have that, whatever it is, inspect.signature should be simple.
For example, if you use kind=VAR_ORDERED_KEYWORD, then it's just a matter of making Parameter.__str__ return whatever-it-is that the docs would have instead of returning the normal **{name}.
This does give a little more weight to proposals that do something
visible in the signature itself--the ***okwargs, or an annotation, or whatever. Makes sense.
But I think all of them have enough down-sides compared to your more favored versions that it wouldn't be nearly enough weight to shift things.
Guido's concern was that in some uncommon cases the difference in performance (both memory and speed) would render OrderedDict undesirable for **kwargs. By storing the initial order on a dict attribute the only overhead would be 1 pointer per dict plus the size of the container.
First, after reading the messages you linked, I don't think your characterizing Guido's concerns fairly.
His first message is not about uncommon cases at all, it's about the fact
It's hard to predict how much of a slowdown a nice C OrderedDict would cause here, and how much less of a slowdown just creating and using __order__ would cause. But from a quick simulation in pure Python, it looks
Perhaps. At this point nothing much has been settled. My gut says you're right, but we'll see. that every function call (at least every one that uses kwargs) will slow down noticeably: "My main concern is speed -- since most code doesn't need it and function calls are already slow (and obviously very common :-) it would be a shame if this slowed down function calls that don't need it noticeably." In the best case the slowdown on each call would be the cost of checking a flag and not even for most calls. In the worst case I image it will be the cost of allocating a tuple/list (a free list would help) and populating it, while again this would only impact functions that define **kwargs). like you're just turning a 2.7x slowdown into a 2.2x slowdown--better, but still clearly noticeable, and therefore not acceptable. At one point I benchmarked my C OrderedDict (issue #16991) against dict. For most operations it was the same, including all non-mutating operations. If I recall correctly, iteration was even faster (due to traversing a linked list rather than a sparse hash table). For mutating operations the overhead resulted in up to a 3x slowdown.
This means you need some way to make sure functions that don't ask for
I think all of your solutions (including storing __kworder__) will be noticeably slower; there's just no way around the fact that maintaining a dict and its order and iterating it in custom order takes longer than
Trying to disguise the ordering info, or make it not quite as slow, or make it opt-out instead of opt-in, etc., doesn't solve this problem. Only finding a way to do nothing at all when nobody wants it solves this
ordered kwargs don't get it (unless you can actually make it not noticeably slower). The fact that this only affects functions that define **kwargs minimizes the scope of the impact. maintaining a dict and iterating it in natural order. I'm pretty sure this only matters in the **kwargs unpacking case. Even then, if we go with OrderedDict unpacking into the new kwargs is only marginally slower than dict. I do not know the impact of unpacking into a secondary order-preserving container in addition to into dict, but I expect it will not be much worse. Again this should only impact cases where the function defines **kwargs. problem. At which point you might as well do the simplest thing--use OrderedDict--for cases where people _do_ want it.
And that comes back to my original point: Where does someone want order?
When they explicitly ask for it, but also when wrapping a function that asks for it in a generic wrapper. And that's the hard part; inside the wrapper (in some library code), you don't know to ask for it; outside the wrapper (in your app), there's no way to do it. I agree that this use case ("perfect kwargs preserving" wrappers) is more complicated under the circumstances you described. However, don't discount the options on the table that do not present those circumstances.
Meanwhile, the "uncommon" cases (which you elsewhere call "rare",
"extreme", and "edge cases") are a different story. I don't think Guido thinks they're very uncommon. He says, "I write code regularly that..." saves kwargs for later, and possibly even piles other stuff into it. And I will argue that while perhaps not uncommon for Guido, they are uncommon for most people.
His primary issue here is that having anything other than a plain dict is wrong and confusing.
Hmm. I did not get that, much less that it is his primary concern. I took away that he was concerned about the performance impact in the case where kwargs outlives the function. Plus, this is Python. As long as it quacks like a dict we're fine. With OrderedDict it's even a subclass of dict.
Secondarily, he also mentions "possible" performance problems. Here is where the memory comes in. Increasing the memory of every kwargs dict by 25% instead of 75% is nice, but I don't think it solves the memory problem, and I don't think the memory problem is the main issue anyway.
I agree it's not the main issue.
This seems to rule out any solution that gives you a modified kwargs dict
I now understand why you proposed the separate __kworder__ magic
when you don't ask for it, whether it's an OrderedDict or a dict with an __order__ attribute or anything else. Due the memory issues? Memory is only an issue in cases where the resultant kwargs is going to be extremely large, which I still say is uncommon. Furthermore, using dict.__order__ means that even then it's an issue only with an extremely large number of kwargs passed in since __order__ wouldn't grow with the dict. And opt-out would allow people to avoid the issue in those cases. parameter solution. But really, given the first problem, any solution that adds _any_ noticeable overhead when you don't ask for it is already ruled out, so trying to put the extra overhead somewhere other than the dict itself doesn't help. You need to actually not create it. There shouldn't be any noticeable overhead so this point is somewhat moot. If there is then this proposal is on pretty shaky ground.
Anyway, after a little more back-burner thought, I actually did think of
At compile time, any code object that has a call to something accessed via a closure cell gets a calls-local flag. (This should catch all decorators and other wrapper-generators, but many false positives as well.) At runtime, when building a function from a code object with the calls-local flag, if any of the __closure__ members are functions with the order-kwargs flag, the new function also gets the flag as well. (This eliminates most of the false positives, without eliminating any actual wrappers unless they're designed pathologically. And if they are
design would mean that all local functions get ordered kwargs, but
some magic that might work. But I don't think you, or anyone else, will like it. I certainly don't. It's good to look at all the options. And this is python-ideas after all. :) pathological, well, then they lose kwargs ordering.) That's all only a couple lines of code in the compiler and interpreter. Tada? Not sure; the idea is too horrible and non-Pythonic to think through all the way. Ultimately, this there's an optimization that usually eliminates the order when you don't need it... which is just wrong. That's not so bad. I don't think something like that would be needed though. This is definitely an implementation detail. It would be important to minimize any further complication to function-call code. Otherwise the implementation will be a hard sell. Sort of relatedly, one thing to consider when adding language features is the impact they will have on other implementations. That relates almost entirely on the explicit additions to the language spec so we need to be careful about favoring solutions strictly for implementation concerns. -eric