[issue18779] Misleading documentations and comments in regular expression about alphanumerics and underscore
New submission from Vajrasky Kok: According to: http://oald8.oxfordlearnersdictionaries.com/dictionary/alphanumeric http://en.wikipedia.org/wiki/Alphanumeric Alphanumeric is defined as [A-Za-z0-9]. Underscore (_) is not one of them. One of the documentation in Python (Doc/tutorial/stdlib2.rst) differentiates them very clearly: "The format uses placeholder names formed by ``$`` with valid Python identifiers (alphanumeric characters and underscores). Surrounding the placeholder with braces allows it to be followed by more alphanumeric letters with no intervening spaces. Writing ``$$`` creates a single escaped ``$``::" Yet, in documentations as well as comments in regex, we implicitely assumes underscore belongs to alphanumeric. Explicit is better than implicit! Attached the patch to differentiate alphanumeric and underscore in documentations and comments in regex. This is important in case someone is confused with this code:
import re re.split('\W', 'haha$hihi*huhu_hehe hoho') ['haha', 'hihi', 'huhu_hehe', 'hoho']
On the side note: In Python code base, sometimes we write "alphanumerics" and "underscores", yet sometimes we write "alphanumeric characters" and "underscore characters". Which one again is the true way? ---------- assignee: docs@python components: Documentation files: fix_alphanumeric_and_underscore_doc_in_regex.patch keywords: patch messages: 195611 nosy: akuchling, docs@python, vajrasky priority: normal severity: normal status: open title: Misleading documentations and comments in regular expression about alphanumerics and underscore versions: Python 3.3, Python 3.4 Added file: http://bugs.python.org/file31371/fix_alphanumeric_and_underscore_doc_in_rege... _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
Changes by Serhiy Storchaka <storchaka@gmail.com>: ---------- components: +Regular Expressions nosy: +ezio.melotti, mrabarnett, pitrou stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
Antoine Pitrou added the comment: I was wondering which doc you were alluding it, before I noticed your patch is against the regex HOWTO. The HOWTO seems quite outdated wrt. Python 3. For example "\w" is not equivalent to "[a-zA-Z0-9_]", anymore, except with the ASCII flag. ---------- title: Misleading documentations and comments in regular expression about alphanumerics and underscore -> Misleading documentations and comments in regular expression HOWTO _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
Vajrasky Kok added the comment: In Lib/re.py, starting from line 77 (Python 3.4): \w Matches any alphanumeric character; equivalent to [a-zA-Z0-9_] in bytes patterns or string patterns with the ASCII flag. In string patterns without the ASCII flag, it will match the range of Unicode alphanumeric characters (letters plus digits plus underscore). With LOCALE, it will match the set [0-9_] plus characters defined as letters for the current locale. The prelude is "Matches any alphanumeric character;". Yet, in any case (bytes, string patterns with ascii flag, string patterns without the ascii flag, strings with locale), the underscore is always included. Then why don't we change the prelude to "Matches any alphanumeric character and underscore character;"? In the description we explain the alphanumeric depending on it's unicode or not can be [A-Za-z0-9] or wider than that. The description is already okay but the prelude is misleading readers. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
R. David Murray added the comment: The answer to the question about "alphanumerics" versus "alphanumeric characters" is that is is mostly likely context-dependent, so I'd have to see particular examples to say which I though read better. So, there is no One True Answer for this question, I think. ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
Changes by A.M. Kuchling <amk@amk.ca>: ---------- nosy: -akuchling _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
A.M. Kuchling added the comment: Unfortunately making the sentences pedantically correct also makes them ungainly, and I think people generally assume that underscores are treated as a letter. ---------- nosy: +akuchling resolution: -> wont fix stage: patch review -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue18779> _______________________________________
participants (5)
-
A.M. Kuchling -
Antoine Pitrou
-
R. David Murray
-
Serhiy Storchaka
-
Vajrasky Kok