Generator objects and list comprehensions?
Hi, Glyph pointed this out to me here: http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html If I do this on Python 3.6:
[(yield 1) for x in range(10)] <generator object <listcomp> at 0x10cd210f8>
If I understand this: https://docs.python.org/3/reference/expressions.html#list-displays then this is a list display and should give a list, not a generator object. Is there a bug in Python, or does the documentation need to be updated? -- Craig
On Wed, Jan 25, 2017 at 4:38 PM, Craig Rodrigues <rodrigc@freebsd.org> wrote:
Glyph pointed this out to me here: http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html
If I do this on Python 3.6:
[(yield 1) for x in range(10)] <generator object <listcomp> at 0x10cd210f8>
If I understand this: https://docs.python.org/3/reference/expressions.html#list-displays then this is a list display and should give a list, not a generator object. Is there a bug in Python, or does the documentation need to be updated?
That looks like an odd interaction between yield expressions and list comps. Since a list comprehension is actually implemented as a nested function, your code actually looks more-or-less like this:
def <listcomp>(iter): result = [] for x in iter: result.append((yield 1)) return result <listcomp>(iter(range(10)) <generator object <listcomp> at 0x10cd210f8>
This function is a generator, and calling it returns what you see above. If you step that iterator, it'll yield 1 ten times, and then raise StopIteration with the resulting list. Based on a cursory examination of the issue at hand, I think what you're working with might be functioning as a coroutine? If so, you may find that using "await" instead of "yield" dodges the problem, as it won't turn the list comp into a generator. But I can't be 100% certain of that. (Also, that would definitely stop you from having single-codebase 2.7/3.x code.) ChrisA
List, set, and dict comprehensions compile like: # input result = [expression for lhs in iterator_expression] # output def comprehension(iterator): out = [] for lhs in iterator: out.append(expression) return out result = comprehension(iter(iterator_expression)) When you make `expression` a `yield` the compiler thinks that `comprehension` is a generator function instead of a normal function. We can manually translate the following comprehension: result = [(yield n) for n in (0, 1)] def comprehension(iterator): out = [] for n in iterator: # (yield n) as an expression is the value sent into this generator out.append((yield n)) return out result = comprehension(iter((0, 1))) We can see this in the behavior of `send` on the resulting generator: In [1]: g = [(yield n) for n in (0, 1)] In [2]: next(g) Out[2]: 0 In [3]: g.send('hello') Out[3]: 1 In [4]: g.send('world') --------------------------------------------------------------------------- StopIteration Traceback (most recent call last) <ipython-input-4-82c589e0c27d> in <module>() ----> 1 g.send('world') StopIteration: ['hello', 'world'] The `return out` gets translated into `raise StopIteration(out)` because the code is a generator. The bytecode for this looks like: In [5]: %%dis ...: [(yield n) for n in (0, 1)] ...: <module> -------- 1 0 LOAD_CONST 0 (<code object <listcomp> at 0x7f4bae68eed0, file "<show>", line 1>) 3 LOAD_CONST 1 ('<listcomp>') 6 MAKE_FUNCTION 0 9 LOAD_CONST 5 ((0, 1)) 12 GET_ITER 13 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 16 POP_TOP 17 LOAD_CONST 4 (None) 20 RETURN_VALUE <module>.<listcomp> ------------------- 1 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) >> 6 FOR_ITER 13 (to 22) 9 STORE_FAST 1 (n) 12 LOAD_FAST 1 (n) 15 YIELD_VALUE 16 LIST_APPEND 2 19 JUMP_ABSOLUTE 6 >> 22 RETURN_VALUE In `<module>` you can see us create the <listcomp> function, call `iter` on `(0, 1)`, and then call `<listcomp>(iter(0, 1))`. In `<listcomp>` you can see the loop like we had above, but unlike a normal comprehension we have a `YIELD_VALUE` (from the `(yield n)`) in the comprehension. The reason that this is different in Python 2 is that list comprehension, and only list comprehensions, compile inline. Instead of creating a new function for the loop, it is done in the scope of the comprehension. For example: # input result = [expression for lhs in iterator_expression] # output result = [] for lhs in iterator_expression: result.append(lhs) This is why `lhs` bleeds out of the comprehension. In Python 2, adding making `expression` a `yield` causes the _calling_ function to become a generator: def f(): [(yield n) for n in range(3)] # no return # py2
type(f()) generator
# py3
type(f()) NoneType
In Python 2 the following will even raise a syntax error: In [5]: def f(): ...: return [(yield n) for n in range(3)] ...: ...: File "<ipython-input-5-3602a9999f46>", line 2 return [(yield n) for n in range(3)] SyntaxError: 'return' with argument inside generator Generator expressions are a little different because the compilation already includes a `yield`. # input (expression for lhs in iterator_expression) # output def genexpr(iterator): for lhs in iterator: yield expression You can actually abuse this to write a cute `flatten` function: `flatten = lambda seq: (None for sub in seq if (yield from sub) and False)` because it translates to: def genexpr(seq): for sub in seq: # (yield from sub) as an expression returns the last sent in value if (yield from sub) and False: # we never enter this branch yield None That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error. On Wed, Jan 25, 2017 at 12:38 AM, Craig Rodrigues <rodrigc@freebsd.org> wrote:
Hi,
Glyph pointed this out to me here: http://twistedmatrix. com/pipermail/twisted-python/2017-January/031106.html
If I do this on Python 3.6:
[(yield 1) for x in range(10)] <generator object <listcomp> at 0x10cd210f8>
If I understand this: https://docs.python.org/ 3/reference/expressions.html#list-displays then this is a list display and should give a list, not a generator object. Is there a bug in Python, or does the documentation need to be updated?
-- Craig
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ joe%40quantopian.com
On 25 January 2017 at 07:01, Chris Angelico <rosuav@gmail.com> wrote:
[(yield 1) for x in range(10)] <generator object <listcomp> at 0x10cd210f8>
This is an old bug, see e.g. http://bugs.python.org/issue10544 The ``yield`` inside comprehensions is bound to the auxiliary function. Instead it should be bound to an enclosing function, like it is done for ``await``. The behaviour of ``await`` in comprehensions is intuitive (since it is simply equivalent to a for-loop):
async def f(i): ... return i
async def g_for(): ... lst = [] ... for i in range(5): ... lst.append(await f(i)) ... print(lst)
g_for().send(None) [0, 1, 2, 3, 4] Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
async def g_comp(): ... print([await f(i) for i in range(5)])
g_comp().send(None) # exactly the same as g_for [0, 1, 2, 3, 4] Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
While current behaviour of ``yield`` in comprehensions is confusing:
def c_for(): ... lst = [] ... for i in range(5): ... lst.append((yield i)) ... print(lst)
c_for().send(None) 0 c_for().send(None) 1 # etc.
def c_comp(): ... print([(yield i) for i in range(5)])
c_comp().send(None) # Naively this should be equivalent to the above, but... <generator object c_comp.<locals>.<listcomp> at 0x7f1fd1faa630> Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'NoneType' object has no attribute 'send'
I promised myself to write a patch, but didn't have time for this yet. I hope I will do this at some point soon. -- Ivan
On 25.01.2017 07:28, Joe Jevnik wrote:
That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error.
Same here; mixing comprehensions and yield (from) can't be explained easily. A SyntaxError would be most appropriate. Regards, Sven
On Wed, Jan 25, 2017 at 6:28 AM, Joe Jevnik <jjevnik@quantopian.com> wrote:
That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error.
Thanks for the very clear explanation.
Would an addition to the documentation for 3.6 giving an abbreviated version help while the devs consider whether any changes are appropriate for 3.7? Steve Holden
On Jan 25, 2017 8:16 AM, "Joe Jevnik" <jjevnik@quantopian.com> wrote: List, set, and dict comprehensions compile like: # input result = [expression for lhs in iterator_expression] # output def comprehension(iterator): out = [] for lhs in iterator: out.append(expression) return out result = comprehension(iter(iterator_expression)) When you make `expression` a `yield` the compiler thinks that `comprehension` is a generator function instead of a normal function. We can manually translate the following comprehension: result = [(yield n) for n in (0, 1)] def comprehension(iterator): out = [] for n in iterator: # (yield n) as an expression is the value sent into this generator out.append((yield n)) return out result = comprehension(iter((0, 1))) We can see this in the behavior of `send` on the resulting generator: In [1]: g = [(yield n) for n in (0, 1)] In [2]: next(g) Out[2]: 0 In [3]: g.send('hello') Out[3]: 1 In [4]: g.send('world') --------------------------------------------------------------------------- StopIteration Traceback (most recent call last) <ipython-input-4-82c589e0c27d> in <module>() ----> 1 g.send('world') StopIteration: ['hello', 'world'] The `return out` gets translated into `raise StopIteration(out)` because the code is a generator. The bytecode for this looks like: In [5]: %%dis ...: [(yield n) for n in (0, 1)] ...: <module> -------- 1 0 LOAD_CONST 0 (<code object <listcomp> at 0x7f4bae68eed0, file "<show>", line 1>) 3 LOAD_CONST 1 ('<listcomp>') 6 MAKE_FUNCTION 0 9 LOAD_CONST 5 ((0, 1)) 12 GET_ITER 13 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 16 POP_TOP 17 LOAD_CONST 4 (None) 20 RETURN_VALUE <module>.<listcomp> ------------------- 1 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) >> 6 FOR_ITER 13 (to 22) 9 STORE_FAST 1 (n) 12 LOAD_FAST 1 (n) 15 YIELD_VALUE 16 LIST_APPEND 2 19 JUMP_ABSOLUTE 6 >> 22 RETURN_VALUE In `<module>` you can see us create the <listcomp> function, call `iter` on `(0, 1)`, and then call `<listcomp>(iter(0, 1))`. In `<listcomp>` you can see the loop like we had above, but unlike a normal comprehension we have a `YIELD_VALUE` (from the `(yield n)`) in the comprehension. The reason that this is different in Python 2 is that list comprehension, and only list comprehensions, compile inline. Instead of creating a new function for the loop, it is done in the scope of the comprehension. For example: # input result = [expression for lhs in iterator_expression] # output result = [] for lhs in iterator_expression: result.append(lhs) This is why `lhs` bleeds out of the comprehension. In Python 2, adding making `expression` a `yield` causes the _calling_ function to become a generator: def f(): [(yield n) for n in range(3)] # no return # py2
type(f()) generator
# py3
type(f()) NoneType
In Python 2 the following will even raise a syntax error: In [5]: def f(): ...: return [(yield n) for n in range(3)] ...: ...: File "<ipython-input-5-3602a9999f46>", line 2 return [(yield n) for n in range(3)] SyntaxError: 'return' with argument inside generator Generator expressions are a little different because the compilation already includes a `yield`. # input (expression for lhs in iterator_expression) # output def genexpr(iterator): for lhs in iterator: yield expression You can actually abuse this to write a cute `flatten` function: `flatten = lambda seq: (None for sub in seq if (yield from sub) and False)` because it translates to: def genexpr(seq): for sub in seq: # (yield from sub) as an expression returns the last sent in value if (yield from sub) and False: # we never enter this branch yield None That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error. Another option would be to restore the py2 behavior by inserting an implicit 'yield from' whenever the embedded expression contains a yield. So in your example where result = [(yield n) for n in (0, 1)] currently gets expanded to result = comprehension(iter((0, 1))) it could notice that the expanded function 'comprehension' is a generator, and emit code like this instead: result = yield from comprehension(iter((0, 1))) At least, this would work for list/dict/set comprehensions; not so much for generator expressions. I assume this is basically how 'await' in comprehensions works in 3.6. I'm not sure this is actually a good idea, given the potential for ambiguity and backwards compatibility considerations -- I'm kind of leaning towards the deprecate/error option on balance :-). But it's an option that would probably be less confusing than the status quo, and would make it easier to write the kind of straddling code the original poster was trying to write. -n
On 26 January 2017 at 00:53, Nathaniel Smith <njs@pobox.com> wrote:
I'm not sure this is actually a good idea, given the potential for ambiguity and backwards compatibility considerations -- I'm kind of leaning towards the deprecate/error option on balance :-). But it's an option that would probably be less confusing than the status quo, and would make it easier to write the kind of straddling code the original poster was trying to write.
Concerning list/set/dict comprehensions, I am much more in favor of making comprehensions simply equivalent to for-loops (more or less like you proposed using yield from). The only reason to introduce auxiliary function scope was to prevent the loop variables from leaking outside comprehensions. Formally, this is indeed backward incompatible, but I doubt many people depend on the current counter-intuitive behavior. Concerning generator expressions, probably it is indeed better to simply prohibit yield inside them. -- Ivan
On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Concerning list/set/dict comprehensions, I am much more in favor of making comprehensions simply equivalent to for-loops (more or less like you proposed using yield from). The only reason to introduce auxiliary function scope was to prevent the loop variables from leaking outside comprehensions. Formally, this is indeed backward incompatible, but I doubt many people depend on the current counter-intuitive behavior.
Concerning generator expressions, probably it is indeed better to simply prohibit yield inside them.
Thank you to everyone who responded to my post and provided excellent analysis. For Python, I don't know what the best way to proceed is: OPTION 1 ======== Make a SyntaxError: [(yield 1) for x in range(10)] and update the documentation to explain that this is an invalid construct. This would have certainly helped me identify the source of the problem as I tried porting buildbot 0.9 to Python 3. However, while not very common, there is Python 2.x code that uses that. I found these cases in the buildbot code which I changed so as to work on Python 2 and 3: https://github.com/buildbot/buildbot/pull/2661 https://github.com/buildbot/buildbot/pull/2673 OPTION 2 ========= Make this return a list on Python 3, like in Python 2: [ (yield 1) for x in range(10)] As pointed out by others on the this mailing list, there are some problems associated with that. I don't know if there are many Python 2 codebases out there with this construct, but it would be nice to have one less Python 2 -> 3 porting gotcha. I'm OK with either approach. Leaving things the way they are in Python 3 is no good, IMHO. -- Craig
On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues <rodrigc@freebsd.org> wrote:
On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Concerning list/set/dict comprehensions, I am much more in favor of making comprehensions simply equivalent to for-loops (more or less like you proposed using yield from). The only reason to introduce auxiliary function scope was to prevent the loop variables from leaking outside comprehensions. Formally, this is indeed backward incompatible, but I doubt many people depend on the current counter-intuitive behavior.
Concerning generator expressions, probably it is indeed better to simply prohibit yield inside them.
Thank you to everyone who responded to my post and provided excellent analysis.
For Python, I don't know what the best way to proceed is:
OPTION 1 ========
Make a SyntaxError: [(yield 1) for x in range(10)] and update the documentation to explain that this is an invalid construct.
This would have certainly helped me identify the source of the problem as I tried porting buildbot 0.9 to Python 3.
However, while not very common, there is Python 2.x code that uses that. I found these cases in the buildbot code which I changed so as to work on Python 2 and 3:
https://github.com/buildbot/buildbot/pull/2661 https://github.com/buildbot/buildbot/pull/2673
OPTION 2 ========= Make this return a list on Python 3, like in Python 2: [ (yield 1) for x in range(10)]
As pointed out by others on the this mailing list, there are some problems associated with that. I don't know if there are many Python 2 codebases out there with this construct, but it would be nice to have one less Python 2 -> 3 porting gotcha.
I'm OK with either approach. Leaving things the way they are in Python 3 is no good, IMHO.
My vote is it be a SyntaxError since you're not getting what you expect from the syntax.
On 30 January 2017 at 19:05, Brett Cannon <brett@python.org> wrote:
On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues <rodrigc@freebsd.org> wrote:
I'm OK with either approach. Leaving things the way they are in Python 3 is no good, IMHO.
My vote is it be a SyntaxError since you're not getting what you expect from the syntax.
I'd agree that's a sensible place for us to end up, as any code relying on the current behaviour is really too clever to be maintainable. In terms of getting there, we'll likely want: - SyntaxWarning or DeprecationWarning in 3.7 - Py3k warning in 2.7.x - SyntaxError in 3.8 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (9)
-
Brett Cannon -
Chris Angelico -
Craig Rodrigues -
Ivan Levkivskyi -
Joe Jevnik -
Nathaniel Smith -
Nick Coghlan -
Steve Holden -
Sven R. Kunze