[Python-ideas] PEP for issue2292, "Missing *-unpacking generalizations"

[Joshua Landau]

The PEP is attached. I'm not sure if I've covered the basics, but it's a try.

If anyone knows how to get the patch (from the bug report) working, or
where to find http://code.python.org/python/users/twouters/starunpack
after code.python.org was deleted in favour of hg.python.org (which
seems not to have it), it'd be nice to know.
-------------- next part --------------
PEP: XXX
Title: Additional Unpacking Generalizations
Version: $Revision$
Last-Modified: $Date$
Author: Joshua Landau <joshua at landau.ws>
Discussions-To: python-ideas at python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Jun-2013
Python-Version: 3.4
Post-History: #TODO


Abstract
========

This PEP proposes extended usages of the ``*`` iterable unpacking operator to allow unpacking in more positions, and an arbitrary number of times, and in several additional circumstances.

Specifically:

Arbitrarily positioned unpacking operators::

    >>> print(*[1], *[2], 3)
    1 2 3
    >>> dict(**{'x': 1}, y=3, **{'z': 2})
    {'x': 1, 'y': 2, 'z': 3}
    >>> def f(*args, last): pass

Keywords arguments must still follow positional arguments but now must also follow ``*``-unpackings. The function of a lone ``*`` is unchanged.

Unpacking is proposed to be allowed inside tuples, lists, sets, dictionaries and comprehensions::

    >>> *range(4), 4
    (0, 1, 2, 3, 4)
    >>> [*range(4), 4]
    [0, 1, 2, 3, 4]
    >>> {*range(4), 4}
    {0, 1, 2, 3, 4}
    >>> {'x': 1, **{'y': 2}}
    {'x': 1, 'y': 2}

    >>> ranges = [range(i) for i in range(5)]
    >>> [*item for item in ranges]
    [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]


Rationale
=========

Current usage of the ``*`` iterable unpacking operator features unnecessary restrictions that can harm readability. There is also asymmetry between what is allowed in assignment unpacking in function definition. This proposal hopes to alleviate a large proportion of this imbalance.

Unpacking multiple times has an obvious rationale. When you want to unpack several iterables into a function definition or follow an unpack with more positional arguments, the most natural way would be to write::

    function(**kw_arguments, **more_arguments)

    function(*arguments, argument)

Simple examples where this is useful are ``print`` and ``str.format``. Instead, you could be forced to write::

    kwargs = dict(kw_arguments)
    kwargs.update(more_arguments)
    function(**kwargs)

    args = list(arguments)
    args.append(arg)
    function(*args)

or, if you know to do so::

    from collections import ChainMap
    function(**ChainMap(more_arguments, arguments))

    from itertools import chain
    function(*chain(args, [arg]))

which add unnecessary line-noise and, with the first methods, causes duplication of work.


Function definitions are also now more symmetrical with assignment; whereas previously just::

    first, *others, last = iterable

was valid, now so too is::

    def f(first, *others, last):
        ...

    f(*iterable)

As PEP 3132 has been finalized, it should already be clear the benefits of this approach. In particular, this should improve the signatures of functions that utilize this feature by moving the unpacking from the body into the definition.


There are two primary rationale for unpacking inside of containers. Firstly there is a symmetry of assignment, where ``fst, *other, lst = elems`` and ``elems = fst, *other, lst`` are approximate inverses, ignoring the specifics of types. This, in effect, simplifies the language by removing special cases.

Secondly, it vastly simplifies types of "addition" such as combining dictionaries, and does so in an unambiguous and well-defined way::

    combination = {**first_dictionary, "x": 1, "y": 2}

instead of::

    combination = first_dictionary.copy()
    combination.update({"x": 1, "y": 2})

which is especially important in contexts where expressions are preferred. This is also useful as a more readable way of summing many lists, such as ``my_list + list(my_tuple) + list(my_range)`` which is now equivalent to just ``[*my_list, *my_tuple, *my_range]``.


A further extension to comprehensions is a logical and necessary extension. It's usage will primarily be a neat replacement for ``[i for j in 2D_list for i in j]``, as the more readable ``[*l for l in 2D_list]``. Other uses are possible, but expected to occur rarely.


Specification
=============

Function calls may accept an unbound number of ``*`` and ``**`` unpackings, which are allowed anywhere that positional and keyword arguments are allowed respectively. In approximate pseudo-notation::

    function(
        argument or *args, argument or *args, ...,
        kwargument or **kwargs, kwargument or **kwargs, ...
    )

As the function ``lambda *args, last: ...`` now does not require ``last`` to be a keyword only argument, ``lambda *args, *, last: ...`` will be valid. No other changes are made to function definition.


Tuples, lists, sets and dictionaries will allow unpacking. This will act as if the elements from unpacked item were inserted in order at the site of unpacking, much as happens in unpacking in a function-call. Dictionaries require ``**`` unpacking, all the others require ``*`` unpacking. A dictionary's key remain in a right-to-left priority order, so ``{**{'a': 1}, 'a': 2, **{'a': 3}}`` evaluates to ``{'a': 3}``.

Comprehensions, by simple extension, will support unpacking. As before, dictionaries require ``**`` unpacking, all the others require ``*`` unpacking and key priorities are unchanged.

Examples include::

    {*[1, 2, 3], 4, 5}

    (*e for e in [[1], [3, 4, 5], [2]])

    {**dictionary for dictionary in (globals(), locals())}

    {**locals(), "override": None}


Disadvantages
=============

Parts of this change are not backwards-compatible.

- ``function(kwarg="foo", *args)`` is no longer valid syntax; ``function(*args, kwarg="foo")`` is required instead

- ``lambda *args, last: ...`` no longer requires ``last`` to be a keyword only argument


Additionally, whilst ``*elements, = iterable`` causes ``elements`` to be a list, ``elements = *iterable,`` causes ``elements`` to be a tuple. The reason for this is may not be obvious at first glance, and may confuse people unfamiliar with the construct.


.. I don't feel I have the standing to make a judgment on these cases. Needless to say the first of these is a more significant hurdle and will affect more working code.


Implementation
==============

An implementation for an old version of Python 3 is found at Issue 2292 on bug tracker [1]_. It has yet to be updated to the most recent Python version.


References
==========

.. [1] Issue 2292, "Missing `*`-unpacking generalizations", Thomas Wouters
   (http://bugs.python.org/issue2292)

.. [2] Discussion on Python-ideas list, "list / array comprehensions extension", Alexander Heger
   (http://mail.python.org/pipermail/python-ideas/2011-December/013097.html)


Copyright
=========

This document has been placed in the public domain.