<div dir="ltr"><div><div><div><div><div>Generators are a wonderful feature of the python language, and one of its best idea.<br><br></div>They are initially very intuitive to understand & easy to use.<br><br></div>However, once you get beyond that; they are actually quite confusing because their behavior is not natural.<br><br>Thus they have a initial easy learning & acceptance curve; and then as you go from initial use to more advanced use, there is a sudden "bump" in the learning curve, which is not as smooth as it could be.<br><br></div>Specifically, the fact that when you call them, they do not actually call your code (but instead call a wrapper) is quite confusing.<br><br></div>Example:<br><br> import __main__<br><br><br> def read_and_prefix_each_line(path):<br> with open(path) as f:<br> data = f.read()<br><br> for s in data.splitlines():<br> yield '!' + s<br><br> def print_prefixed_file(path):<br> reader = read_and_prefix_each_line(path) #LINE 12<br><br> print('Here is how %r looks prefixed' % path)<br><br> for s in reader: #LINE 16<br> print(s)<br><br> print_prefixed_file(__main__.__file__)<br> print_prefixed_file('nonexistent')<br><br></div>Will produce the following:<br><br><div style="margin-left:40px">Traceback (most recent call last):<br> File "x.py", line 20, in <module><br> print_prefixed_file('nonexistent')<br> File "x.py", line 16, in print_prefixed_file<br> for s in reader:<br> File "x.py", line 5, in read_and_prefix_each_line<br> with open(path) as f:<br>IOError: [Errno 2] No such file or directory: 'nonexistent'<br></div><div><br></div><div>This is quite confusing to a person who has been using generators for a month, and thinks they understand them.</div><div><br></div><div>WHY is the traceback happening at line 16 instead of at line 12, when the function is called?<br><br></div><div>It is much more intuitive, and natural [to a beginner], to expect the failure to open the file "nonexistent" to happen at line 12, not line 16.</div><div><br></div><div>So, now, the user, while trying to figure out a bug, has to learn that:</div><div><ul><li>NO, calling a generator (which looks like a function) does not actually call the body of the function (as the user defined it)</li><li>Instead it calls some generator wrapper.</li><li>And finally the first time the __next__ method of the wrapper is called, the body of the function (as the user defined it) gets called.</li></ul><p>And this learning curve is quite steep.</p><p>It is made even harder by the following:</p><p> >>> def f(): yield 1<br> ...<br> >>> f<br> <function f at 0x7f748507c5f0><br></p><p>So the user is now convinced that 'f' really is a function.</p><p>Further investigation makes it even more confusing:</p><p> >>> f()<br> <generator object f at 0x7f74850716e0><br></p><p>At this point, the user starts to suspect that something is kind of unusual about 'yield' keyword.</p><p>Eventually, after a while, the user starts to figure this out:</p><p> >>> def f(): print('f started'); yield 1<br> ...<br> >>> f()<br> <generator object f at 0x7f3a0baf5fc0><br> >>> f().__next__()<br> f started<br> 1<br> >>><br></p><p>And eventually after reading <a href="https://docs.python.org/3/reference/datamodel.html">https://docs.python.org/3/reference/datamodel.html</a> the following sentence:</p><p>"The following flag bits are defined for <code class="gmail-xref gmail-py gmail-py-attr gmail-docutils gmail-literal"><span class="gmail-pre">co_flags</span></code>: bit <code class="gmail-docutils gmail-literal"><span class="gmail-pre">0x04</span></code> is set if
the function uses the <code class="gmail-docutils gmail-literal"><span class="gmail-pre">*arguments</span></code> syntax to accept an arbitrary number of
positional arguments; bit <code class="gmail-docutils gmail-literal"><span class="gmail-pre">0x08</span></code> is set if the function uses the
<code class="gmail-docutils gmail-literal"><span class="gmail-pre">**keywords</span></code> syntax to accept arbitrary keyword arguments; bit <code class="gmail-docutils gmail-literal"><span class="gmail-pre">0x20</span></code> is set
if the function is a generator."</p><p>Finally figures it out:</p><p> >>> def f(): yield 1<br> ...<br> >>> f<br> <function f at 0x7f73f38089d8><br> >>> f.__code__.co_flags & 0x20<br> 32<br></p><p>My point is ... this learning process is overly confusing to a new user & not a smooth learning curve.</p><p>Here is, just a quick initial proposal on how to fix it:</p> >>> def f(): yield 1</div><div> >>> ...</div><div> Syntax Error: the 'yield' keyword can only be used in a generator; please be sure to use @generator before the definition of the generator<br></div><div><br></div><div> >>> @generator<br> ... def f(): yield 1<br> ...<br> >>> f<br> <generator f at 0x7f73f38089d8></div><div><br></div><div>Just the fact it says '<generator f at ...' instead of '<function f at ...' would be a *BIG* help to starting users.</div><div><br></div><div>This would also really drive home, from the start, the idea to the user, that:</div><div><ul><li>A generator is special, is not a function</li><li>Calling the generator does not call the generator, but its wrapper. (the generator is not called until its __next_ method is called)<br></li></ul></div><div><br></div><div>Next:</div><div><br></div><div><ol><li>Don't make this the default; but require: from __future__ import generator to activate this feature (for the next few releases of python)</li><li>Regarding contexts, *REQUIRE* an argument to generator that tells it how to have the generator interact with contexts. I.E.: Something like:</li></ol><p style="margin-left:80px">@generator(catpure_context_at_start = true)<br></p><p style="margin-left:80px">def f(): yield 1<br></p><p style="margin-left:40px">With three options: (a) capture context at start; (b) capture context on first call to __next__; (c) don't capture context at all, but have it work the natural way, you expressed two emails ago (i.e.: each time use the context of the caller, not a special context for the generator).<br></p><p style="margin-left:40px">Finally if a user attempts to user a generator with contexts but without one of these three parameters, throw a syntax error & tell the user the context usage must be specified.</p><p>The major point of all this, is make the learning curve easier for users, so generators are:<br></p><ul><li>Intuitive, easy to pick up & quick use (as they currently are)</li><li>Intuitive easy to pick up & quick to use, as you go from a beginner to a medium level user (making it easier to learn their specific in's & out's)</li><li>Intuitive, easy to pick up & quick to use, as you go from medium user to advanced user (and need to make them interact in different way with contexts, etc).<br></li></ul><p><br></p><p><br></p></div><div><p><br></p><p><br></p><p><br></p><p><br></p></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Oct 15, 2017 at 9:33 AM, Paul Moore <span dir="ltr"><<a href="mailto:p.f.moore@gmail.com" target="_blank">p.f.moore@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 15 October 2017 at 13:51, Amit Green <<a href="mailto:amit.mixie@gmail.com">amit.mixie@gmail.com</a>> wrote:<br>
> Once again, I think Paul Moore gets to the heart of the issue.<br>
><br>
> Generators are simply confusing & async even more so.<br>
><br>
> Per my earlier email, the fact that generators look like functions, but are<br>
> not functions, is at the root of the confusion.<br>
<br>
</span>I don't agree. I don't find generators *at all* confusing. They are a<br>
very natural way of expressing things, as has been proven by how<br>
popular they are in the Python community.<br>
<br>
I don't *personally* understand async, but I'm currently willing to<br>
reserve judgement until I've been in a situation where it would be<br>
useful, and therefore needed to learn it.<br>
<span class="HOEnZb"><font color="#888888"><br>
Paul<br>
</font></span></blockquote></div><br></div>