<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">On Wed, Sep 6, 2017 at 8:16 PM, Guido van Rossum </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-m_-4002427021569365499gmail-">On Wed, Sep 6, 2017 at 8:07 AM, Koos Zevenhoven <span dir="ltr"><<a href="mailto:k7hoven@gmail.com" target="_blank">k7hoven@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span></span>I think yield from should have the same semantics as iterating over the generator with next/send, and PEP 555 has no issues with this.</div></blockquote><div><br></div></span><div>I think the onus is on you and Greg to show a realistic example that shows why this is necessary.</div><div><br></div></div></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace">​Well, regarding this part, it's just that things like</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">for obj in gen:</div><div class="gmail_default" style="font-family:monospace,monospace">​    yield obj</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">often get modernized into</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">yield from gen</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">And realistic examples of that include pretty much any normal use of yield from.<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>So far all the argumentation about this has been of the form "if you have code that currently does this (example using foo) and you refactor it in using yield from (example using bar), and if you were relying on context propagation back out of calls, then it should still propagate out."</div><div><br></div></div></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace">​So here's a realistic example, with the semantics of PEP 550 applied to a decimal.setcontext() kind of thing, but it could be anything using var.set(value):</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">def process_data_buffers(​buffers)<wbr>:</div><div class="gmail_default" style="font-family:monospace,monospace">    setcontext(default_context)</div><div class="gmail_default" style="font-family:monospace,monospace">    for buf in buffers:</div><div class="gmail_default" style="font-family:monospace,monospace">        for data in buf:</div><div class="gmail_default" style="font-family:monospace,monospace">            if data.tag == "NEW_PRECISION":</div><div class="gmail_default" style="font-family:monospace,monospace">                setcontext(context_based_on(<wbr>data))</div><div class="gmail_default" style="font-family:monospace,monospace">            else:</div><div class="gmail_default" style="font-family:monospace,monospace">                yield compute(data)</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Code smells? Yes, but maybe you often see much worse things, so let's say it's fine.</div><div class="gmail_default" style="font-family:monospace,monospace"> <br></div><div class="gmail_default" style="font-family:monospace,monospace">​But then, if you refactor it into a subgenerator like this:<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><div class="gmail_default">def process_data_buffer(buffer):</div><div class="gmail_default">    for data in buf:<br></div><div class="gmail_default"><div class="gmail_default">        if data.tag == "NEW_PRECISION":</div><div class="gmail_default">            setcontext(context_based_on(<wbr>data))</div><div class="gmail_default">        else:</div><div class="gmail_default">            yield compute(data)</div></div><div class="gmail_default"><br></div><div class="gmail_default">def process_data_buffers(​buffers)<wbr>:</div><div class="gmail_default">    setcontext(default_context)</div><div class="gmail_default">    for buf in buffers:</div><div class="gmail_default">        yield from buf</div><div class="gmail_default"><br></div><div><br></div><div>Now, if setcontext uses PEP 550 semantics, the refactoring broke the code, because a generator introduce a scope barrier by adding a LogicalContext on the stack, and setcontext is only local to the process_data_buffer subroutine. But the programmer is puzzled, because with regular functions it had worked just fine in a similar situation before they learned about generators:</div><div><br></div><div><br></div><div><div class="gmail_default"><div class="gmail_default">def process_data_buffer(buffer, output):</div><div class="gmail_default"><div class="gmail_default">    for data in buf:</div><div class="gmail_default">        if data.tag == "precision change":</div><div class="gmail_default">            setcontext(context_based_on(<wbr>data))</div><div class="gmail_default">        else:</div><div class="gmail_default">            output.append(compute(data))</div></div><div class="gmail_default"><br></div><div class="gmail_default">def process_data_buffers(​buffers)<wbr>:</div><div class="gmail_default">    output = []</div><div class="gmail_default">    setcontext(default_context)</div><div class="gmail_default">    for buf in buffers:</div><div class="gmail_default">        process_data_buffer(buf, output)</div></div></div></div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace">​In fact, this code had another problem, namely that the context state is leaked out of process_d​ata_buffers, because PEP 550 leaks context state out of functions, but not out of generators. But we can easily imagine that the unit tests for process_data_buffers *do* pass. </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">But let's look at a user of the functionality:</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">def get_total():</div><div class="gmail_default" style="font-family:monospace,monospace">    return sum(process_data_buffers(get_buffers()))</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">setcontext(somecontext)<br></div><div class="gmail_default" style="font-family:monospace,monospace">value = get_total() * compute_factor()<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Now the code is broken, because setcontext(somecontext) has no effect, because get_total() leaks out another context. Not to mention that our data buffer source now has control over the behavior of compute_factor(). But if one is lucky, the last line was written as</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">value = compute_factor() * get_total()</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">And hooray, the code works!</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">(Except for perhaps the code that is run after this.)</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Now this was of course a completely fictional example, and hopefully I didn't introduce any bugs or syntax errors other than the ones I described. I haven't seen code like this anywhere, but somehow we caught the problems anyway.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">-- Koos</div></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><div><br></div>-- <br><div class="gmail-m_-4002427021569365499gmail_signature">+ Koos Zevenhoven + <a href="http://twitter.com/k7hoven" target="_blank">http://twitter.com/k7hoven</a> +</div>
</div></div>