[Python-ideas] How assignment should work with generators?

Brendan Barnwell brenbarn at brenbarn.net
Thu Nov 30 00:25:25 EST 2017

On 2017-11-29 20:43, Steven D'Aprano wrote:
> At the point that you are conjuring from thin air an invisible suitcase
> that is an exact clone of the original suitcase, in order to unpack the
> clone without disturbing the original, I think the metaphor is so far
> from the real-world unpacking of suitcases that it no longer applies.

	It is not an exact clone of the original suitcase, because the original 
suitcase is a collection with stable contents (i.e., cannot be 
exhausted), but the "clone" (the iterator) CAN be exhausted.  It 
iterates over the same *values*, but that doesn't mean it's the same thing.

> Besides, it's not even correct to say that an invisible suitcase
> (iterator) is constructed.
> # Python 3.5
> py> dis.dis("a, b, c = [97, 98, x]")
>    1           0 LOAD_CONST               0 (97)
>                3 LOAD_CONST               1 (98)
>                6 LOAD_NAME                0 (x)
>                9 ROT_THREE
>               10 ROT_TWO
>               11 STORE_NAME               1 (a)
>               14 STORE_NAME               2 (b)
>               17 STORE_NAME               3 (c)
>               20 LOAD_CONST               2 (None)
>               23 RETURN_VALUE
> Before iterators existed, Python had sequence unpacking going back to at
> least Python 1.5 if not older, so even if Python used a temporary and
> invisible iterator *now* that has not always been the case and it might
> not be the case in the future or in alternate interpreters.
> Even if Python *sometimes* builds a temporary and invisible iterator, it
> doesn't do it all the time, and when it does, it is pure implementation,
> not interface. The interpreter is free to do whatever it likes behind
> the scenes, but there's no iterator involved in the high-level Python
> statement:
>      a, b, c = [1, 2, 3]
> That code involves a list and three assignment targets, that is all.

	The code only has a list and three assignment targets, but that doesn't 
mean that that's all it "involves".  The expression "a + b" only has two 
variables and a plus sign, but it involves a call to __add__ which is 
not explicitly represented.  Things like this aren't implementation 
details.  Indeed, they're precisely the opposite: they are a high level 
specification of an API for how syntax is converted into semantics.

>> This is the same as the behavior for "for"
>> loops: if you do "for item in [1, 2, 3]", the actual thing you're
>> unrolling is an iterator over the list.
> No, the actual thing *I* am unrolling is exactly what I write in my
> code, which is the list [1, 2, 3].
> I don't care what the Python interpreter iterates over internally, so
> long as the results are the same. It can use an iterator, or unroll the
> loop at compile-time, or turn it into recursion over a linked list for
> all I care.
> As much as possible, we should avoid thinking about implementation
> details when trying to understand high-level semantics of Python code.
> Otherwise, our mental models become obsolete when the implementation
> changes.

	Don't you see a bit of irony in arguing based on the compiled bytecode, 
and then saying you don't care about implementation details? :-)  Here 
is a simpler example:

class Foo(object):
     def __iter__(self):
         print("You tried to call iter on me!")

 >>> a, b, c = Foo()
You tried to call iter on me!
Traceback (most recent call last):
   File "<pyshell#2>", line 1, in <module>
     a, b, c = Foo()
TypeError: iter() returned non-iterator of type 'NoneType'

	You can see that iter() is called, even though "exactly what I wrote in 
the code" is not iter(Foo()) but just Foo().  The "implementation 
detail" is that this function call is concealed within a bytecode called 
"UNPACK_SEQUENCE".  Another implementation detail is that in your 
example that bytecode not used, but that's only because you decompiled 
an expression with a literal list.  If you do "x = [1, 2, 3]" and then 
decompile "a, b, c = x", you will see the UNPACK_SEQUENCE bytecode.

	These details of the bytecode are implementation details.  What is not 
an implementation detail is the iteration protocol, which is exactly the 
kind of high-level semantic thing you're talking about.  The iteration 
protocol says that when you go to iterate over something, iter() is 
called on it, and then next() is called on the result of that call.

	Because of this, I am loath to pretend that whether "a, b, c = x" is 
"like unpacking a suitcase" depends on whether x happens to be a list, 
some other iterable, or some other iterator.  The end result in all 
cases is that the thing that actually doles out the items is an 
iterator.  Sometimes that iterator is connected to a stable base (some 
kind of collection) that can be re-iterated; sometimes it isn't.  But 
that doesn't change the iteration protocol.  The interpreter is not free 
to do what it likes behind the scenes; an implementation that did not 
call __iter__ in the above case would be errroneous.  __iter__ is part 
of the interface of any type that defines it.

Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

More information about the Python-ideas mailing list