A sorted version of **kwargs
Hi there python-ideas! Does it bother anyone else that it's so cumbersome to instantiate an OrderedDict with ordered data? >>> from collections import OrderedDict >>> OrderedDict(b=1,a=2) OrderedDict([('a', 2), ('b', 1)]) # Order lost. Boooo. >>> OrderedDict([('b',1),('a',2)]) OrderedDict([('b', 1), ('a', 2)]) Obviously, OrderedDict's __init__ method (like all other functions) never gets a chance to see the kwargs dict in the order it was specified. It's usually faked by accepting the sequence of (key, val) tuples, as above. I personally think it would be nice to be able to ask the interpreter to keep track of the order of the arguments to my function, something like: def sweet_function_name(*args, **kwargs, ***an_odict_of_kwargs): pass I'm not married to the syntax. What do you think about the idea?
On 20 January 2011 04:29, Don Spaulding <donspauldingii@gmail.com> wrote:
Hi there python-ideas! Does it bother anyone else that it's so cumbersome to instantiate an OrderedDict with ordered data? >>> from collections import OrderedDict >>> OrderedDict(b=1,a=2) OrderedDict([('a', 2), ('b', 1)]) # Order lost. Boooo. >>> OrderedDict([('b',1),('a',2)]) OrderedDict([('b', 1), ('a', 2)]) Obviously, OrderedDict's __init__ method (like all other functions) never gets a chance to see the kwargs dict in the order it was specified. It's usually faked by accepting the sequence of (key, val) tuples, as above. I personally think it would be nice to be able to ask the interpreter to keep track of the order of the arguments to my function, something like: def sweet_function_name(*args, **kwargs, ***an_odict_of_kwargs): pass I'm not married to the syntax. What do you think about the idea?
FYI this was discussed before on this list at least once: http://mail.python.org/pipermail/python-ideas/2009-April/004163.html -- Arnaud
On Thu, Jan 20, 2011 at 1:21 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
On 20 January 2011 04:29, Don Spaulding <donspauldingii@gmail.com> wrote:
Hi there python-ideas! Does it bother anyone else that it's so cumbersome to instantiate an OrderedDict with ordered data? >>> from collections import OrderedDict >>> OrderedDict(b=1,a=2) OrderedDict([('a', 2), ('b', 1)]) # Order lost. Boooo. >>> OrderedDict([('b',1),('a',2)]) OrderedDict([('b', 1), ('a', 2)]) Obviously, OrderedDict's __init__ method (like all other functions) never gets a chance to see the kwargs dict in the order it was specified. It's usually faked by accepting the sequence of (key, val) tuples, as above. I personally think it would be nice to be able to ask the interpreter to keep track of the order of the arguments to my function, something like: def sweet_function_name(*args, **kwargs, ***an_odict_of_kwargs): pass I'm not married to the syntax. What do you think about the idea?
FYI this was discussed before on this list at least once:
http://mail.python.org/pipermail/python-ideas/2009-April/004163.html
-- Arnaud
So it was. Thanks for that link. Am I to assume nothing ever came of that discussion?
Don Spaulding wrote:
Obviously, OrderedDict's __init__ method (like all other functions) never gets a chance to see the kwargs dict in the order it was specified. It's usually faked by accepting the sequence of (key, val) tuples, as above. I personally think it would be nice to be able to ask the interpreter to keep track of the order of the arguments to my function, something like:
def sweet_function_name(*args, **kwargs, ***an_odict_of_kwargs): pass
I'm not married to the syntax. What do you think about the idea?
I would be +0 on making **kwargs an ordered dict automatically, and -1 on adding ***ordered_kwargs. Because kwargs is mostly used only for argument passing, and generally with only a small number of items, it probably doesn't matter too much if it's slightly slower than an unordered dict. But adding a third asterisk just makes it ugly, and it leaves open the question what happens when you try to mix **kw and ***okw in the same function call, as you do above. -- Steven
On Thu, Jan 20, 2011 at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I would be +0 on making **kwargs an ordered dict automatically, and -1 on adding ***ordered_kwargs. Because kwargs is mostly used only for argument passing, and generally with only a small number of items, it probably doesn't matter too much if it's slightly slower than an unordered dict.
Yeah, simply making the kwargs dict always ordered is likely the way we would do it. That's also the only solution with any chance of working by default with the way most decorators are structured (accepting *args and **kwargs and passing them to the wrapped function). To expand on Raymond's response in the previous thread on this topic, there are likely a number of steps to this process: 1. Provide a _collections.OrderedDict C implementation 2. Create a PEP to gain agreement from other implementations (especially IronPython, PyPy and Jython) to proceed with the remaining steps 3. Make it a builtin class (odict?) in its own right (with collections.OrderedDict becoming an alias for the builtin type) 4. Update the interpreter to use the new builtin type for kwargs containers Use various microbenchmarks to check that the use of the new odict builtin type instead of a plain dict doesn't slow things down too much. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
On Thu, Jan 20, 2011 at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I would be +0 on making **kwargs an ordered dict automatically, and -1 on adding ***ordered_kwargs. Because kwargs is mostly used only for argument passing, and generally with only a small number of items, it probably doesn't matter too much if it's slightly slower than an unordered dict.
Yeah, simply making the kwargs dict always ordered is likely the way we would do it. That's also the only solution with any chance of working by default with the way most decorators are structured (accepting *args and **kwargs and passing them to the wrapped function).
-1. How often do you really need this ? In which of those cases wouldn't a static code analysis give you the call order of the parameters already ? "Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 20 2011)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Jan 20, 2011 at 6:05 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Nick Coghlan wrote:
On Thu, Jan 20, 2011 at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I would be +0 on making **kwargs an ordered dict automatically, and -1 on adding ***ordered_kwargs. Because kwargs is mostly used only for argument passing, and generally with only a small number of items, it probably doesn't matter too much if it's slightly slower than an unordered dict.
Yeah, simply making the kwargs dict always ordered is likely the way we would do it. That's also the only solution with any chance of working by default with the way most decorators are structured (accepting *args and **kwargs and passing them to the wrapped function).
-1.
How often do you really need this ?
In which of those cases wouldn't a static code analysis give you the call order of the parameters already ?
"Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object.
What he says. In addition, I wonder what the semantics would be if the caller passed **d where d was an *unordered* dict... -- --Guido van Rossum (python.org/~guido)
On Thu, Jan 20, 2011 at 6:42 PM, Guido van Rossum <guido@python.org> wrote:
-1.
How often do you really need this ?
In which of those cases wouldn't a static code analysis give you the call order of the parameters already ?
"Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object.
What he says.
In addition, I wonder what the semantics would be if the caller passed **d where d was an *unordered* dict...
What if the default behavior stays as it is today, but a magic decorator is added, (maybe @ordered_kwargs or some such), and only for these kind of functions the new behavior applies. Also, given such a decorator, when given **d where d is a regular dict, the implementation could possibly throw an error. (Or maybe it is up to the implementor of the specific function). Cheers, Imri -- Imri Goldberg -------------------------------------- http://plnnr.com/ - automatic trip planning http://www.algorithm.co.il/blogs/ -------------------------------------- -- insert signature here ----
On Thu, Jan 20, 2011 at 8:42 AM, Guido van Rossum <guido@python.org> wrote:
On Thu, Jan 20, 2011 at 6:05 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Nick Coghlan wrote:
On Thu, Jan 20, 2011 at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I would be +0 on making **kwargs an ordered dict automatically, and -1 on adding ***ordered_kwargs. Because kwargs is mostly used only for argument passing, and generally with only a small number of items, it probably doesn't matter too much if it's slightly slower than an unordered dict.
Yeah, simply making the kwargs dict always ordered is likely the way we would do it. That's also the only solution with any chance of working by default with the way most decorators are structured (accepting *args and **kwargs and passing them to the wrapped function).
-1.
How often do you really need this ?
In which of those cases wouldn't a static code analysis give you the call order of the parameters already ?
"Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object.
What he says.
In addition, I wonder what the semantics would be if the caller passed **d where d was an *unordered* dict...
Wouldn't this be a good argument for the original proposal? That there wouldn't be confusion about whether you were getting an odict or a dict with ***? Also, would functions that didn't specify this behavior see an actual performance hit? I assumed that given the existence of METH_NOARGS and friends that there was some kind of optimization going on here, but I can't get it to turn up on timeit. Geremy Condra
On Thu, Jan 20, 2011 at 11:42 AM, Guido van Rossum <guido@python.org> wrote: ..
In addition, I wonder what the semantics would be if the caller passed **d where d was an *unordered* dict...
Presumably, the caller would know whether the called function is sensitive to the order of keyword arguments and will use odict when it matters. The function called as say f(x=1, y=2, **d) will initialize its internal keyword odict with an equivalent of kwds = odict([('x', 1), ('y', 2)]); kwd.update(d.items()) and exhibit undefined behavior if d is unordered and f depends on the order of keywords.
On Thu, Jan 20, 2011 at 8:42 AM, Guido van Rossum <guido@python.org> wrote:
On Thu, Jan 20, 2011 at 6:05 AM, M.-A. Lemburg <mal@egenix.com> wrote:
"Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object.
What he says.
In addition, I wonder what the semantics would be if the caller passed **d where d was an *unordered* dict...
-- --Guido van Rossum (python.org/~guido)
Agree with both. And if we were to make this change, my next thought is that what I really want is an ordered multi-set, since in some scenarios where I want ordered parameters I also want repeated parameters. I don't think we should go there. Back to the original problem though: if the issue is that creating an ordered dict is clumsy and perhaps interfering with adoption and usage then perhaps the notation for ordered dict could be improved. Just as we can now use {...} for both dicts and sets, perhaps we could add [ 'b' : 1, 'a' : 2 ] as a more convenient way of writing OrderedDict([('b', 1), ('a', 2)]) This is parallel to the way that [1,2] is an ordered container while {1,2} is unordered. --- Bruce Latest blog post: http://www.vroospeak.com/2010/12/fix-filibuster.html Learn about security: http://j.mp/gruyere-security
On 21 January 2011 06:13, Bruce Leban <bruce@leapyear.org> wrote:
Back to the original problem though: if the issue is that creating an ordered dict is clumsy and perhaps interfering with adoption and usage then perhaps the notation for ordered dict could be improved. Just as we can now use {...} for both dicts and sets, perhaps we could add
[ 'b' : 1, 'a' : 2 ]
as a more convenient way of writing
OrderedDict([('b', 1), ('a', 2)])
This is parallel to the way that [1,2] is an ordered container while {1,2} is unordered.
['b':1] would then be ambiguous (appears to be a slice of a list). More obvious in the case of [1:2] ... Personally, I'm of the opinion that if an actual dictionary or subclass is passed via **kw, the same type of dictionary should be used for the keyword arguments i.e.: test(**dict(a=1, b=2)) => unordered dict passed. test(**odict(a=1, b=2)) => ordered dict passed. The only difficulty then is passing parameters into the ordered dict constructor in the desired order - I can't think of a reasonable way of doing that. Good old chicken and egg problem. Tim Delaney
On Thu, Jan 20, 2011 at 11:47 AM, Tim Delaney <timothy.c.delaney@gmail.com>wrote:
['b':1] would then be ambiguous (appears to be a slice of a list). More obvious in the case of [1:2] ...
We use parenthesis for tuples and avoid the ambiguity by writing (1,). In the same way, we could require your examples to be written ['b':1,] and [1:2,] --- Bruce
On Thu, Jan 20, 2011 at 2:05 PM, Bruce Leban <bruce@leapyear.org> wrote:
On Thu, Jan 20, 2011 at 11:47 AM, Tim Delaney <timothy.c.delaney@gmail.com
wrote:
['b':1] would then be ambiguous (appears to be a slice of a list). More obvious in the case of [1:2] ...
We use parenthesis for tuples and avoid the ambiguity by writing (1,). In the same way, we could require your examples to be written ['b':1,] and [1:2,]
Please, not this. I like the idea of syntactic support for the odict, but no need to spread the (foo,) syntax around. It's too easy to misinterpret when you do a quick scan of a body of code.
On 20/01/2011 19:47, Tim Delaney wrote:
On 21 January 2011 06:13, Bruce Leban <bruce@leapyear.org <mailto:bruce@leapyear.org>> wrote:
Back to the original problem though: if the issue is that creating an ordered dict is clumsy and perhaps interfering with adoption and usage then perhaps the notation for ordered dict could be improved. Just as we can now use {...} for both dicts and sets, perhaps we could add
[ 'b' : 1, 'a' : 2 ]
as a more convenient way of writing
OrderedDict([('b', 1), ('a', 2)])
This is parallel to the way that [1,2] is an ordered container while {1,2} is unordered.
['b':1] would then be ambiguous (appears to be a slice of a list). More obvious in the case of [1:2] ...
[snip] In what way is it ambiguous? [1] isn't ambiguous, is it? spam[1] is subscripting and [1] is a list; spam[1 : 2] is slicing and [1 : 2] would be an ordered dict.
On Thu, Jan 20, 2011 at 2:47 PM, Tim Delaney <timothy.c.delaney@gmail.com> wrote: ..
['b':1] would then be ambiguous (appears to be a slice of a list). More obvious in the case of [1:2] ...
x[1:2], x[1:2,], and x[1:2, 3:4] are all valid syntaxes. (NumPy uses the latter for multi-dimensional slicing.) However, I don't see an ambiguity here. We don't have an ambiguity between tuple syntax and function calls: (a, b, c) vs. f(a, b, c).
On 2011-01-20, at 17:42 , Guido van Rossum wrote:
On Thu, Jan 20, 2011 at 6:05 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Nick Coghlan wrote:
On Thu, Jan 20, 2011 at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I would be +0 on making **kwargs an ordered dict automatically, and -1 on adding ***ordered_kwargs. Because kwargs is mostly used only for argument passing, and generally with only a small number of items, it probably doesn't matter too much if it's slightly slower than an unordered dict.
Yeah, simply making the kwargs dict always ordered is likely the way we would do it. That's also the only solution with any chance of working by default with the way most decorators are structured (accepting *args and **kwargs and passing them to the wrapped function).
-1.
How often do you really need this ?
In which of those cases wouldn't a static code analysis give you the call order of the parameters already ?
"Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object.
What he says.
In addition, I wonder what the semantics would be if the caller passed **d where d was an *unordered* dict… Create an ordereddict based on d's iteration order would seem logical (but order would keep conserved for kwargs passed in explicitly, so in `foo(a=some, b=other, c=stuff, **d)` where `d` is a `dict` a, b, c would be the first three keys and then the keys of d would be stuffed after that in whatever order happens).
Do Python 3's semantics allow for further kwargs after **kwargs?
On Fri, Jan 21, 2011 at 2:42 AM, Guido van Rossum <guido@python.org> wrote:
On Thu, Jan 20, 2011 at 6:05 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Nick Coghlan wrote:
Yeah, simply making the kwargs dict always ordered is likely the way we would do it. That's also the only solution with any chance of working by default with the way most decorators are structured (accepting *args and **kwargs and passing them to the wrapped function).
-1.
How often do you really need this ?
In which of those cases wouldn't a static code analysis give you the call order of the parameters already ?
"Nice to have" is not good enough to warrant a slow down of all function calls involving keyword arguments, adding overhead for other Python implementations and possibly causing problems with 3rd party extensions relying on getting a PyDict for the keyword arguments object.
What he says.
I actually agree as well, but I was misremembering how the construction of the kwargs dict worked and hence was thinking that was the only possible way this could work (since the interpreter didn't know anything about the target function while building the dict). Actually checking with dis.dis and looking at the associated code in ceval.c corrected my misapprehension (the function is actually retrieved first, while the kwargs are still on the stack in the appropriate order, so it is theoretically possible for the function to influence how the args are stored). So, as an alternative proposal, perhaps it would be possible to add a new protocol that allowed a callable to flag that an ordered dictionary should be used for kwargs (e.g. an "__ordered_call__" boolean attribute, with C level flags to speed up the common builtin callable cases). A @functools.ordered_call decorator could then just do "__ordered_call__ = True" to set the flag appropriately. (You could also be even more flexible and devise a protocol that supported any type for kwargs, but I think that would just be far more complicated without a corresponding increase in expressive power) Ordinary calls that used "x=y" or "**d" would be slowed down marginally due to the check for the new attribute on the supplied callable, but that impact should be minimal (especially for the builtin cases, which would just be checking a C struct field) and other calls would be entirely unaffected. The CPython specific impact would largely be limited to update_keyword_args() and implementing C level fields with associated __ordered_call__ properties on the various builtin callable objects. There would also need to be a new variant of PyEval_EvalCodeEx that either used an ordered dictionary for kwdict, or else allowed kwdict to be created and passed in by the calling code rather than implicitly created via PyDict_New(). A builtin version of collections.OrderedDict would still be a precursor to this idea though, so creating a _collections.OrderedDict C implementation still sounds like the right starting point for anyone that is particularly keen to see this idea progress. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (13)
-
Alexander Belopolsky -
Arnaud Delobelle -
Bruce Leban -
Don Spaulding -
geremy condra -
Guido van Rossum -
Imri Goldberg -
M.-A. Lemburg -
Masklinn -
MRAB -
Nick Coghlan -
Steven D'Aprano -
Tim Delaney