[issue15457] consistent treatment of generator terminology
New submission from Chris Jerdonek <chris.jerdonek@gmail.com>: The documentation on generators (outside of PEP 255) does not currently educate the reader on the more specific "generator function" and "generator iterator" terminology, or at least not in any consistent or systematic way. For example, the glossary includes only a single entry for "generator," and that entry does not mention the two more specific forms. I think it would help for general discourse purposes if this distinction were made clearer, while still continuing to allow for the use of the generic word "generator" when the context makes it clear. There are also cases where index entries can be improved in this regard, and where references to the section containing details about generators can still be added. I am in the process of completing a proposed patch. ---------- assignee: docs@python components: Documentation keywords: easy messages: 166474 nosy: cjerdonek, docs@python priority: normal severity: normal status: open title: consistent treatment of generator terminology versions: Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Chris Jerdonek <chris.jerdonek@gmail.com> added the comment: Attaching patch. ---------- keywords: +patch Added file: http://bugs.python.org/file26526/issue-15457-1.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Changes by Nick Coghlan <ncoghlan@gmail.com>: ---------- nosy: +ncoghlan _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: I think the current entry calling a generator function a generator is wrong, so I appreciate your patch. Generator functions return instances of class 'generator'*. I personally refer to generators as generator-iterators about as often as I refer to lists as list-sequences. (Generators implement the iterator protocol; lists implement the sequence protocol.) * I personally think of a generator function as defining a virtual subclass of class generator and returning instances of that (vitual) subclass. The gf does this# by setting the gi_code attribute of instances to the code object compiled from its body. (Or perhaps it calls the generator class with its code object as parameter. In any case, this is what the special gf.__call__ method does instead of executing its code attribute. The call protocol makes this magic easy.) The generator subclass, then, is all instances with the same gi_code attribute. The name of the 'subclass' is the name of the generator function, which is also g.gi_code.co_name. # Although 'gi_code' does not have a leading underscore, I am not sure whether it is intended to be public or if the mechanism is CPython specific. It *is' the only way to get the gf/subclass name from the generator. What I think still needs correction is "generator then controls the execution of a generator function". The generator function has already run to create the generator. What is controlled is the execution of the code object compiled from the body of a gf. And the 'generator function' in question is not just 'a generator function' but 'the generator function that created the generator'. So I suggest as more correct something like "generator then controls the execution of the [code object compiled from the] body of the generator function that created it" The part in [] is true but might be omitted if it seems too much in the context. ---------- nosy: +terry.reedy versions: +Python 2.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Chris Jerdonek <chris.jerdonek@gmail.com> added the comment: Thanks a lot for the excellent feedback. I'll update the patch to incorporate it. One background note: I tried not to be too strict about using "generator iterator" or "generator function" in place of "generator" in the general docs in deference to this remark in PEP 255: "Note that when the intent is clear from context, the unqualified name "generator" may be used to refer either to a generator-function or a generator-iterator." ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Chris Jerdonek <chris.jerdonek@gmail.com> added the comment: Attaching updated patch incorporating feedback. It would be nice to have a good term for "the generator function code object associated with a generator" to distinguish it from the original generator function as you point out. There are many places in the docs where this distinction is blurred, for example throughout the subsection describing generator methods: "generator.__next__(): Starts the execution of a generator function or resumes it at the last executed yield expression..." http://docs.python.org/dev/reference/expressions.html#generator.__next__ By the way, I think it would be a good idea to add "code object" to the glossary as it is not currently listed. In particular, the entry could link to the section of the docs that discuss code objects in more detail: http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy specifically: http://docs.python.org/dev/reference/datamodel.html#index-52 ---------- Added file: http://bugs.python.org/file26552/issue-15457-2.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: Up until today, I had assumed that it was the generator.__next__ method that was associated with the compiled body. But in writing my response, I investigated and discovered
def gf(): yield None
gf().gi_code is gf.__code__ True
Then i realized that the simple call protocal -- a callable is an object with an executable __call__ method -- makes the magic simpler than I thought. Generator functions must have a special __call__ method that simply creates and returns a generator instance with the code object attached (instead of executing the code). Since code objects are referred to in various places (compile(), exec(), probably def statement doc), I agree that there should be a minimal glossary entry. One can't really understand generator functions and generators without knowing that def statements create both a function object and an attached code object, but that they can operate somewhat separately. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Terry J. Reedy <tjreedy@udel.edu> added the comment: I will try to read your new patch when I am fresher. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Chris Jerdonek <chris.jerdonek@gmail.com> added the comment: I created issue 15476 to add "code object" to the glossary.
Generator functions must have a special __call__ method that simply creates and returns a generator instance with the code object attached (instead of executing the code).
Just to be sure, is this the same as what you were saying in your previous comment? "The gf does this# by setting the gi_code attribute of instances to the code object compiled from its body. (Or perhaps it calls the generator class with its code object as parameter. In any case, this is what the special gf.__call__ method does instead of executing its code attribute." Since you're not sure whether the gi_code attribute is meant to be public, it may be best not to mention it explicitly in the docs. This is consistent with what I have done in the latest patch. Thanks in advance for reviewing the update. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Chris Jerdonek added the comment: Note to reviewers: changing to "needs patch" because I want to make changes to the latest patch before this is reviewed (e.g. to the index directives). I should be able to do this by the end of the weekend. ---------- stage: -> needs patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Chris Jerdonek added the comment: This slipped under my radar. After starting work on this again, I realize that it would make sense to complete issue 15476 first. The reason is that it would improve the process of referencing "code object" in the generator docs (Terry's suggestion) once "code object" is itself properly documented. ---------- dependencies: +Add "code object" to glossary _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Changes by Terry J. Reedy <tjreedy@udel.edu>: ---------- versions: +Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue15457> _______________________________________
Cheryl Sabella <cheryl.sabella@gmail.com> added the comment: Issue24400 changed some of the documentation wording around generators. I don't know if there is still interest in applying the other parts of this patch. ---------- nosy: +cheryl.sabella versions: +Python 3.7, Python 3.8 -Python 2.7, Python 3.3, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue15457> _______________________________________
participants (4)
-
Cheryl Sabella -
Chris Jerdonek
-
Nick Coghlan
-
Terry J. Reedy