[issue8968] token type constants are not documented
New submission from Ilya Sandler <ilya.sandler@gmail.com>: the token module defines constants for token types e.g NAME = 1 NUMBER = 2 STRING = 3 NEWLINE = 4 etc. These constants are very useful for any code which needs to tokenize python source, yet they are not listed in the documentation. Is this a documentation bug? ---------- assignee: docs@python components: Documentation messages: 107509 nosy: docs@python, isandler priority: normal severity: normal status: open title: token type constants are not documented versions: Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Éric Araujo <merwok@netwok.org> added the comment: It is. Constants should be documented in the regular doc, not only by introspection or reading the source. See the doc for tokenize: “All constants from the token module are also exported from tokenize, as are two additional token type values”. Thanks for the report. Would you mind writing a patch? ---------- nosy: +merwok versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2 -Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Ilya Sandler <ilya.sandler@gmail.com> added the comment: I'm attaching a documentation patch. Do note that there is also a bit of code-level inconsistency: a few tokens (COMMENT, NL) are defined in tokenize module which is strange and inconvenient. Should that be fixed too? (by moving token definitions into token module) ---------- keywords: +patch Added file: http://bugs.python.org/file17641/token.rst.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Georg Brandl <georg@python.org> added the comment: Five comments: * I would list them all in one directive, like this: .. data:: FOO BAR BAZ which makes the display more compact. * There is no description in that directive. Best move part of the description above them in it. * NL and COMMENT are defined in tokenize because they are neither defined nor used by the Python tokenizer. ---------- nosy: +georg.brandl _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Ilya Sandler <ilya.sandler@gmail.com> added the comment:
* I would list them all in one directive, like this:
Ok, done. Attaching the updated patch
There is no description in that directive. Best move part of the description above them in it.
I am not sure I understand your request. Could you clarify?
NL and COMMENT are defined in tokenize because they are neither defined nor used by the Python tokenizer.
Oh, I just realized the source of my misunderstanding: token.py captures token types as they are returned by python's own tokenizer. While tokenize module does its own tokenization and produces results which are a bit different. COMMENT and NL tokens is one of the differences but there is a difference in how the operation tokens are treated. In fact tokenize's docstring explicitly says so. ... It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators ... I think this clarification should go into official docs as well, would you agree? PS. even with this clarification, I find the situation quite a bit confusing: especially given that tok_name dict from token module does have a name for COMMENT and that name is inserted by tokenize module.. So I still feel that just adding COMMENT and NL to token module (with clarification that they are not generated by the python's own tokenizer) would be a bit cleaner. ---------- Added file: http://bugs.python.org/file17652/token.rst.patch.v2 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Georg Brandl <georg@python.org> added the comment: Committed in r85614. ---------- resolution: -> fixed status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Éric Araujo <merwok@netwok.org> added the comment: It is. Constants should be documented in the regular doc, not only by introspection or reading the source. See the doc for tokenize: “All constants from the token module are also exported from tokenize, as are two additional token type values”. Thanks for the report. Would you mind writing a patch? ---------- nosy: +merwok versions: +Python 2.6, Python 2.7, Python 3.1, Python 3.2 -Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Ilya Sandler <ilya.sandler@gmail.com> added the comment: I'm attaching a documentation patch. Do note that there is also a bit of code-level inconsistency: a few tokens (COMMENT, NL) are defined in tokenize module which is strange and inconvenient. Should that be fixed too? (by moving token definitions into token module) ---------- keywords: +patch Added file: http://bugs.python.org/file17641/token.rst.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Georg Brandl <georg@python.org> added the comment: Five comments: * I would list them all in one directive, like this: .. data:: FOO BAR BAZ which makes the display more compact. * There is no description in that directive. Best move part of the description above them in it. * NL and COMMENT are defined in tokenize because they are neither defined nor used by the Python tokenizer. ---------- nosy: +georg.brandl _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Ilya Sandler <ilya.sandler@gmail.com> added the comment:
* I would list them all in one directive, like this:
Ok, done. Attaching the updated patch
There is no description in that directive. Best move part of the description above them in it.
I am not sure I understand your request. Could you clarify?
NL and COMMENT are defined in tokenize because they are neither defined nor used by the Python tokenizer.
Oh, I just realized the source of my misunderstanding: token.py captures token types as they are returned by python's own tokenizer. While tokenize module does its own tokenization and produces results which are a bit different. COMMENT and NL tokens is one of the differences but there is a difference in how the operation tokens are treated. In fact tokenize's docstring explicitly says so. ... It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators ... I think this clarification should go into official docs as well, would you agree? PS. even with this clarification, I find the situation quite a bit confusing: especially given that tok_name dict from token module does have a name for COMMENT and that name is inserted by tokenize module.. So I still feel that just adding COMMENT and NL to token module (with clarification that they are not generated by the python's own tokenizer) would be a bit cleaner. ---------- Added file: http://bugs.python.org/file17652/token.rst.patch.v2 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
Georg Brandl <georg@python.org> added the comment: Committed in r85614. ---------- resolution: -> fixed status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue8968> _______________________________________
participants (3)
-
Georg Brandl -
Ilya Sandler
-
Éric Araujo