<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Sep 6, 2017 at 1:39 PM, Koos Zevenhoven <span dir="ltr"><<a href="mailto:k7hoven@gmail.com" target="_blank">k7hoven@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">On Wed, Sep 6, 2017 at 8:16 PM, Guido van Rossum </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div></span><div class="gmail_extra"><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-3753583963370598501gmail-m_-4002427021569365499gmail-">On Wed, Sep 6, 2017 at 8:07 AM, Koos Zevenhoven <span dir="ltr"><<a href="mailto:k7hoven@gmail.com" target="_blank">k7hoven@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span></span>I think yield from should have the same semantics as iterating over the generator with next/send, and PEP 555 has no issues with this.</div></blockquote><div><br></div></span><div>I think the onus is on you and Greg to show a realistic example that shows why this is necessary.</div><div><br></div></div></div></div></blockquote><div><br></div></span><div><div style="font-family:monospace,monospace">Well, regarding this part, it's just that things like</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">for obj in gen:</div><div style="font-family:monospace,monospace"> yield obj</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">often get modernized into</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">yield from gen</div></div></div></div></div></blockquote><div><br></div><div>I know that that's the pattern, but everybody just shows the same foo/bar example.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div style="font-family:monospace,monospace">And realistic examples of that include pretty much any normal use of yield from.<br></div></div></div></div></div></blockquote><div><br></div><div>There aren't actually any "normal" uses of yield from. The vast majority of uses of yield from are in coroutines written using yield from.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div style="font-family:monospace,monospace"><br></div></div><span class=""><div style="font-family:monospace,monospace"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>So far all the argumentation about this has been of the form "if you have code that currently does this (example using foo) and you refactor it in using yield from (example using bar), and if you were relying on context propagation back out of calls, then it should still propagate out."</div><div><br></div></div></div></div></blockquote><div><br></div></span><div><div style="font-family:monospace,monospace">So here's a realistic example, with the semantics of PEP 550 applied to a decimal.setcontext() kind of thing, but it could be anything using var.set(value):</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">def process_data_buffers(buffers)<wbr>:</div><div style="font-family:monospace,monospace"> setcontext(default_context)</div><div style="font-family:monospace,monospace"> for buf in buffers:</div><div style="font-family:monospace,monospace"> for data in buf:</div><div style="font-family:monospace,monospace"> if data.tag == "NEW_PRECISION":</div><div style="font-family:monospace,monospace"> setcontext(context_based_on(da<wbr>ta))</div><div style="font-family:monospace,monospace"> else:</div><div style="font-family:monospace,monospace"> yield compute(data)</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Code smells? Yes, but maybe you often see much worse things, so let's say it's fine.</div><div style="font-family:monospace,monospace"> <br></div><div style="font-family:monospace,monospace">But then, if you refactor it into a subgenerator like this:<br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><div>def process_data_buffer(buffer):</div><div> for data in buf:<br></div><div><div> if data.tag == "NEW_PRECISION":</div><div> setcontext(context_based_on(da<wbr>ta))</div><div> else:</div><div> yield compute(data)</div></div><div><br></div><div>def process_data_buffers(buffers)<wbr>:</div><div> setcontext(default_context)</div><div> for buf in buffers:</div><div> yield from buf</div><div><br></div><div><br></div><div>Now, if setcontext uses PEP 550 semantics, the refactoring broke the code, because a generator introduce a scope barrier by adding a LogicalContext on the stack, and setcontext is only local to the process_data_buffer subroutine. But the programmer is puzzled, because with regular functions it had worked just fine in a similar situation before they learned about generators:</div><div><br></div><div><br></div><div><div><div>def process_data_buffer(buffer, output):</div><div><div> for data in buf:</div><div> if data.tag == "precision change":</div><div> setcontext(context_based_on(da<wbr>ta))</div><div> else:</div><div> output.append(compute(data))</div></div><div><br></div><div>def process_data_buffers(buffers)<wbr>:</div><div> output = []</div><div> setcontext(default_context)</div><div> for buf in buffers:</div><div> process_data_buffer(buf, output)</div></div></div></div><br></div><div><div style="font-family:monospace,monospace">In fact, this code had another problem, namely that the context state is leaked out of process_data_buffers, because PEP 550 leaks context state out of functions, but not out of generators. But we can easily imagine that the unit tests for process_data_buffers *do* pass. </div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">But let's look at a user of the functionality:</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">def get_total():</div><div style="font-family:monospace,monospace"> return sum(process_data_buffers(get_<wbr>buffers()))</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">setcontext(somecontext)<br></div><div style="font-family:monospace,monospace">value = get_total() * compute_factor()<br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Now the code is broken, because setcontext(somecontext) has no effect, because get_total() leaks out another context. Not to mention that our data buffer source now has control over the behavior of compute_factor(). But if one is lucky, the last line was written as</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">value = compute_factor() * get_total()</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">And hooray, the code works!</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">(Except for perhaps the code that is run after this.)</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Now this was of course a completely fictional example, and hopefully I didn't introduce any bugs or syntax errors other than the ones I described. I haven't seen code like this anywhere, but somehow we caught the problems anyway.</div><span class="HOEnZb"></span></div></div></div></div></blockquote></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">Yeah, so my claim this is simply a non-problem, and you've pretty much just proved that by failing to come up with pointers to actual code that would suffer from this. Clearly you're not aware of any such code.<br clear="all"></div><div class="gmail_extra"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div></div>