[Python-ideas] PEP for issue2292, "Missing *-unpacking generalizations"

[Joshua Landau]

On 8 July 2013 01:56, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> On 07.07.2013 01:03, Joshua Landau wrote:
>
>>     Function calls may accept an unbound number of ``*`` and ``**``
>> unpackings. Arguments can now occur in any position in a function
>> call. As usual, keyword arguments always go to their respective keys
>> and positional arguments are then placed into the remaining positional
>> slots. In approximate pseudo-notation::
>>
>>         function(
>>             argument or keyword_argument or *args or **kwargs,
>>             argument or keyword_argument or *args or **kwargs,
>>             ...
>
>
> What do you exactly mean by "remaining positional slots"? Please note
> that the current behaviour is to raise TypeError when several (more
> than 1) arguments match the same parameter slot. IMHO it must be kept.

You're right -- I've never gotten that error before, so this is
actually new to me. That is a nicer solution, and it keeps things
clean.

> Another question is related to this matter as well: if we adopt
> the idea of more than one **kwargs in function call -- what about
> key duplication? I.e. whether:
>
>     fun(**{'a': 1}, **{'a': 2})
>
> ...should raise TypeError as well, or should it be equivalent to
> fun(a=2)?
>
> My first thought was that it should raise TypeError -- prohibition
> of parameter duplication is a simple and well settled rule for Python
> function calls.  On second thought: it could be relaxed a bit if we
> agreed about another rule that would be simple enough, e.g.: "for
> anything *after* the first '**kwargs' (or maybe also bare '**,'?)
> another rule is applied: later arguments override earlier (looking
> from left to right), as in dict(...)/.update(...) or as in
> {**foo, **bar} in literals (if the rest of the PEP is accepted).

My first opinion would be that if relaxation is something people find
useful, it would be suited to a separate proposal; it seems outside of
this PEP's scope á mon avis.

Given:

>>> {1:"original", 1:"override"}
{1: 'override'}

the most consistent behaviours would be what are in the PEP already,
and I think that's worth keeping.


---


Thinking about examples, the two cases ("status quo" rules [1] and
relaxed rules [2]) would allow things like:

def f(a, b, c=0, d=0, e=0): ...

"Status Quo" rules:

    f(a, e=e, d=d, *[b, c])

Relaxed rules only:

    f(a, e=e, d=d, b, c)


I brought up the idea for the Relaxed rules because the priority rules
for arguments are somewhat complicated when you add in the ability to
have multiple *args and **kwargs, and remove the restriction of *args
after positionals and **kwargs after positionals.

However, considering that the Relaxed rules are never actually useful
AFAICT (there's no real reason to define positionals after keywords),
this would be a simplification to the specification alone. That'll
make it easier to learn the rules, I believe, but simply saying "write
your arguments in a sane order" should do more than enough to cover it
anyway.

Personally, the rule from the issue itself (positionals, then
keywords) is the simplest, but I agree with Guido that it's not worth
breaking backwards compatibility. In a sense, then, the best way to
describe the "Status Quo" as:

Positionals, then Keywords -- but *if you must* you are allowed to put
"*args" after keywords.


I'm still undecided, so I'll leave this for others to comment on. An
updated version of the PEP that removes the changes to function
definitions and discusses the alternatives for function calls is
attached. I haven't double-checked it, so it may be a bit rougher
around the edges.
-------------- next part --------------
PEP: XXX
Title: Additional Unpacking Generalizations
Version: $Revision$
Last-Modified: $Date$
Author: Joshua Landau <joshua at landau.ws>
Discussions-To: python-ideas at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Jun-2013
Python-Version: 3.4
Post-History: #TODO


Abstract
========

This PEP proposes extended usages of the ``*`` iterable unpacking operator to allow unpacking in more positions, and an arbitrary number of times, and in several additional circumstances.

Specifically:

Arbitrarily positioned unpacking operators::

    >>> print(*[1], *[2], 3)
    1 2 3
    >>> dict(**{'x': 1}, y=3, **{'z': 2})
    {'x': 1, 'y': 2, 'z': 3}

Function calls currently have the restriction that keyword arguments must follow positional arguments and ``**`` unpackings must additionally follow ``*`` unpackings. Because of the new levity for ``*`` and ``**`` unpackings, it may be advisable to list some or all of these restritions.

As currently, if an argument is given multiple times - such as a positional argument given both positionally and by keyword - a TypeError is raised.


Unpacking is proposed to be allowed inside tuples, lists, sets, dictionaries and comprehensions::

    >>> *range(4), 4
    (0, 1, 2, 3, 4)
    >>> [*range(4), 4]
    [0, 1, 2, 3, 4]
    >>> {*range(4), 4}
    {0, 1, 2, 3, 4}
    >>> {'x': 1, **{'y': 2}}
    {'x': 1, 'y': 2}

    >>> ranges = [range(i) for i in range(5)]
    >>> [*item for item in ranges]
    [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]


Rationale
=========

Current usage of the ``*`` iterable unpacking operator features unnecessary restrictions that can harm readability.

Unpacking multiple times has an obvious rationale. When you want to unpack several iterables into a function definition or follow an unpack with more positional arguments, the most natural way would be to write::

    function(**kw_arguments, **more_arguments)

    function(*arguments, argument)

Simple examples where this is useful are ``print`` and ``str.format``. Instead, you could be forced to write::

    kwargs = dict(kw_arguments)
    kwargs.update(more_arguments)
    function(**kwargs)

    args = list(arguments)
    args.append(arg)
    function(*args)

or, if you know to do so::

    from collections import ChainMap
    function(**ChainMap(more_arguments, arguments))

    from itertools import chain
    function(*chain(args, [arg]))

which add unnecessary line-noise and, with the first methods, causes duplication of work.


There are two primary rationale for unpacking inside of containers. Firstly there is a symmetry of assignment, where ``fst, *other, lst = elems`` and ``elems = fst, *other, lst`` are approximate inverses, ignoring the specifics of types. This, in effect, simplifies the language by removing special cases.

Secondly, it vastly simplifies types of "addition" such as combining dictionaries, and does so in an unambiguous and well-defined way::

    combination = {**first_dictionary, "x": 1, "y": 2}

instead of::

    combination = first_dictionary.copy()
    combination.update({"x": 1, "y": 2})

which is especially important in contexts where expressions are preferred. This is also useful as a more readable way of summing many lists, such as ``my_list + list(my_tuple) + list(my_range)`` which is now equivalent to just ``[*my_list, *my_tuple, *my_range]``.


The addition of unpacking to comprehensions is a logical extension. It's usage will primarily be a neat replacement for ``[i for j in 2D_list for i in j]``, as the more readable ``[*l for l in 2D_list]``. Other uses are possible, but expected to occur rarely.


Specification
=============

Function calls may accept an unbound number of ``*`` and ``**`` unpackings.

Function calls currently have the restriction that keyword arguments must follow positional arguments and ``**`` unpackings must additionally follow ``*`` unpackings. Because of the new levity for ``*`` and ``**`` unpackings, it may be advisable to list some or all of these restritions.

As currently, if an argument is given multiple times - such as a positional argument given both positionally and by keyword - a TypeError is raised.

If the restrictions are kept, a function call will look like this::

	function(
	    argument or *args, argument or *args, ...,
	    kwargument or *args, kwargument or *args, ...,
	    kwargument or **kwargs, kwargument or **kwargs, ...
	)

If they are removed completely, a function call will look like this::
    
    function(
        argument or keyword_argument or `*`args or **kwargs,
        argument or keyword_argument or `*`args or **kwargs,
        ...
    )


Tuples, lists, sets and dictionaries will allow unpacking. This will act as if the elements from unpacked item were inserted in order at the site of unpacking, much as happens in unpacking in a function-call. Dictionaries require ``**`` unpacking, all the others require ``*`` unpacking. A dictionary's key remain in a right-to-left priority order, so ``{**{'a': 1}, 'a': 2, **{'a': 3}}`` evaluates to ``{'a': 3}``.

Comprehensions, by simple extension, will support unpacking. As before, dictionaries require ``**`` unpacking, all the others require ``*`` unpacking and key priorities are unchanged.

Examples include::

    {*[1, 2, 3], 4, 5}

    (*e for e in [[1], [3, 4, 5], [2]])

    {**dictionary for dictionary in (globals(), locals())}

    {**locals(), "override": None}


Disadvantages
=============

If the current restrictions for function call arguments (keyword arguments must follow positional arguments and ``**`` unpackings must additionally follow ``*`` unpackings) are kept, the allowable orders for arguments in a function call is more complicated than before. The simplest explanation for the rules may be "positional arguments come first and keyword arguments follow, but ``*`` unpackings are allowed after keyword arguments".

If the current restrictions are lifted, there are no obvious gains to code as the only new orders that are allowed look silly: ``f(a, e=e, d=d, b, c)`` being a simpler example.


Whilst ``*elements, = iterable`` causes ``elements`` to be a list, ``elements = *iterable,`` causes ``elements`` to be a tuple. The reason for this is may not be obvious at first glance, and may confuse people unfamiliar with the construct.


Implementation
==============

An implementation for an old version of Python 3 is found at Issue 2292 on bug tracker [1]_, although several changes should be made.

Several changes should be made:

- It has yet to be updated to the most recent Python version

- It features a now redundant replacement for "yield from" which should be removed

- It also loses support for calling function with keyword arguments before positional arguments, which is an unnecessary backwards-incompatible change

- If the restrictions on the order of arguments in a function call are partially or fully lifted, they would need to be included


References
==========

.. [1] Issue 2292, "Missing `*`-unpacking generalizations", Thomas Wouters
   (http://bugs.python.org/issue2292)

.. [2] Discussion on Python-ideas list, "list / array comprehensions extension", Alexander Heger
   (http://mail.python.org/pipermail/python-ideas/2011-December/013097.html)


Copyright
=========

This document has been placed in the public domain.