[Python-ideas] Fwd: Fwd: Fwd: unpacking generalisations for list comprehension

[Chris Angelico]

On Fri, Oct 21, 2016 at 6:09 AM, Random832 <random832 at fastmail.com> wrote:
> On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
>> Hi, I contributed the current list comprehension implementation (when
>> refactoring it for Python 3 to avoid leaking the iteration variable,
>> as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar
>> for a series of nested for and if statements" is precisely my
>> understanding of how they work, and what they mean.
>
> But it's simply not true. It has never been true and it will never be
> true. Something is not "syntactic sugar" if it doesn't compile to the
> exact same sequence of operations as the thing it is supposedly
> syntactic sugar for. It's a useful teaching tool (though you've
> eventually got to teach the differences), but claiming that it's
> "syntactic sugar" - and "non-negotiably" so, at that - implies that it
> is a literal transformation.

But it is. There are two caveats to the transformation: firstly, it's
done in a nested function (as of Py3), and secondly, the core
operations are done with direct opcodes rather than looking up the
".append" method; but other than that, yes, it's exactly the same.
Here's the disassembly (in 2.7, to avoid the indirection of the nested
function):

>>> def f1(x):
...     return [ord(ch) for ch in x]
...
>>> dis.dis(f1)
  2           0 BUILD_LIST               0
              3 LOAD_FAST                0 (x)
              6 GET_ITER
        >>    7 FOR_ITER                18 (to 28)
             10 STORE_FAST               1 (ch)
             13 LOAD_GLOBAL              0 (ord)
             16 LOAD_FAST                1 (ch)
             19 CALL_FUNCTION            1
             22 LIST_APPEND              2
             25 JUMP_ABSOLUTE            7
        >>   28 RETURN_VALUE
>>> def f2(x):
...     ret = []
...     for ch in x:
...         ret.append(ord(ch))
...     return ret
...
>>> dis.dis(f2)
  2           0 BUILD_LIST               0
              3 STORE_FAST               1 (ret)

  3           6 SETUP_LOOP              33 (to 42)
              9 LOAD_FAST                0 (x)
             12 GET_ITER
        >>   13 FOR_ITER                25 (to 41)
             16 STORE_FAST               2 (ch)

  4          19 LOAD_FAST                1 (ret)
             22 LOAD_ATTR                0 (append)
             25 LOAD_GLOBAL              1 (ord)
             28 LOAD_FAST                2 (ch)
             31 CALL_FUNCTION            1
             34 CALL_FUNCTION            1
             37 POP_TOP
             38 JUMP_ABSOLUTE           13
        >>   41 POP_BLOCK

  5     >>   42 LOAD_FAST                1 (ret)
             45 RETURN_VALUE
>>>


Okay, so what exactly is going on here? Looks to me like there's some
optimization happening in the list comp, but you can see that the same
code is being emitted. It's not *perfectly* identical, but that's
mainly because CPython doesn't take advantage of the fact that 'ret'
was initialized to a list - it still does the full "look up 'append',
then call it" work. I'm not sure why SETUP_LOOP exists in the full
version and not the comprehension, but I believe it's to do with the
break and continue keywords, which can't happen inside a
comprehension. So, again, it's optimizations that are possible in the
comprehension, but otherwise, the code is identical.

Maybe "syntactic sugar" is pushing it a bit, but there's no
fundamental difference between the two. Imagine if an optimizing
compiler could (a) notice that there's no use of break/continue, and
(b) do some static type analysis to see that 'ret' is always a list
(and not a subclass thereof), and optimize the multi-line version. At
that point, the two forms would look almost, or maybe completely,
identical. So I'd support the "syntactic sugar" label here.

Why is this discussion still on python-ideas? Shouldn't it be on
python-demanding-explanations-for-status-quo by now?

ChrisA