[issue28489] Fix comment in tokenizer.c
New submission from Ryan Gonzalez: Attached is a little fix for a comment in tokenizer.c. I noticed that it was never updated for the inclusion of f-strings. ---------- assignee: docs@python components: Documentation files: 0001-Fix-comment-in-tokenizer.c.patch keywords: patch messages: 279056 nosy: Ryan.Gonzalez, docs@python priority: normal severity: normal status: open title: Fix comment in tokenizer.c versions: Python 3.6 Added file: http://bugs.python.org/file45156/0001-Fix-comment-in-tokenizer.c.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Stéphane Wirtel added the comment: Thank you for this patch. ---------- nosy: +matrixise versions: +Python 3.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Eric V. Smith added the comment: I'd actually change this to be something like: Process b"", r"", u"", and the various legal combinations. There are 24 total combinations when you add upper case. I actually wrote a script in Lib/tokenize.py to generate them all. And when I add binary f-strings, that number climbs to 80. ---------- nosy: +eric.smith _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Eric V. Smith added the comment: FWIW, in Lib/tokenize.py, it's _all_string_prefixes():
_all_string_prefixes() set(['', 'FR', 'rB', 'rF', 'BR', 'Fr', 'RF', 'rf', 'RB', 'fr', 'B', 'rb', 'F', 'Br', 'R', 'U', 'br', 'fR', 'b', 'f', 'Rb', 'Rf', 'r', 'u', 'bR'])
My basic point is that trying to list them all is hard and a maintenance problem. So as long as we're not being exhaustive, the comment should just state the gist of what the code does. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Ryan Gonzalez added the comment: @eric.smith How's this instead? ---------- Added file: http://bugs.python.org/file45160/0001-Fix-comment-in-tokenizer.c.patch _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Ryan Gonzalez added the comment: Ugh, hit Submit too soon. I meant to say the patch that has 20:13 as the date (I should've probably given them different names...). ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Eric V. Smith added the comment: That's great. Thanks! ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Changes by Eric V. Smith <eric@trueblade.com>: ---------- stage: -> commit review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Serhiy Storchaka added the comment: There are just 8 legal combinations if ignore case:
import tokenize sorted({x.lower() for x in tokenize._all_string_prefixes() if x}) ['b', 'br', 'f', 'fr', 'r', 'rb', 'rf', 'u']
---------- nosy: +serhiy.storchaka _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Eric V. Smith added the comment: Right, that's basically what _all_string_prefixes() does: it starts with the 6 unique prefixes that are case- and order- independent ('b', 'r', 'u', 'f', 'br', 'fr'), and adds the cased and ordered versions. If you're saying that we should list those 8, and say "with all combinations of case", then I think we'd better off listing the 6 and saying "with all combinations of case and order". That's mainly because if "fbr" gets added, then the list of ordered ones gets larger. But it's just a comment. I think we should just commit Ryan's last patch. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Serhiy Storchaka added the comment: No, I'm just saying that the list of combinations is not so large. Ryan's patch LGTM. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Roundup Robot added the comment: New changeset 8b20ed083a94 by Berker Peksag in branch '3.6': Issue #28489: Fix comment in tokenizer.c https://hg.python.org/cpython/rev/8b20ed083a94 New changeset 72ec2c599301 by Berker Peksag in branch 'default': Issue #28489: Merge from 3.6 https://hg.python.org/cpython/rev/72ec2c599301 ---------- nosy: +python-dev _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Berker Peksag added the comment: I've applied Ryan's patch since it's been sitting in the 'commit review' queue. Thanks! ---------- nosy: +berker.peksag resolution: -> fixed stage: commit review -> resolved status: open -> closed type: -> behavior _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Roundup Robot added the comment: New changeset bc2b2193520cea1adc44d27c9f7721fc6ad37293 by Berker Peksag in branch '3.6': Issue #28489: Fix comment in tokenizer.c https://github.com/python/cpython/commit/bc2b2193520cea1adc44d27c9f7721fc6ad... ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
Roundup Robot added the comment: New changeset bc2b2193520cea1adc44d27c9f7721fc6ad37293 by Berker Peksag in branch 'master': Issue #28489: Fix comment in tokenizer.c https://github.com/python/cpython/commit/bc2b2193520cea1adc44d27c9f7721fc6ad... New changeset 9babb676a3c143099ab548ccc72c1ee897335e84 by Berker Peksag in branch 'master': Issue #28489: Merge from 3.6 https://github.com/python/cpython/commit/9babb676a3c143099ab548ccc72c1ee8973... ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue28489> _______________________________________
participants (6)
-
Berker Peksag -
Eric V. Smith
-
Roundup Robot
-
Ryan Gonzalez
-
Serhiy Storchaka
-
Stéphane Wirtel