[issue38566] Description of '\w' behavior is vague in `re` documentation
New submission from James Gerity <snoopjedi@gmail.com>: The documentation for the `re` library¹ describes the behavior of the specifier '\w' as matching "Unicode word characters," which is very vague. The closest thing I can find that corresponds to this language is the guidance offered in Unicode Technical Standard #18², which defines the class `<word_character>` to include all alphabetic and decimal codepoints, as well as U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER. This does not appear to be a correct description of `re`, however, as these zero-width characters are not counted when matching '\w', e.g.: ```
re.match('\w*', 'Auf\u200Clage') <re.Match object; span=(0, 3), match='Auf'>
It seems from examining the CPython source³ that SRE treats '\w' as meaning any alphanumeric character OR U+005F SPACING UNDERSCORE, which does not match any Unicode class definition I've been able to find.
Can anyone provide clarification on what part of Unicode this documentation is referring to? If there is some other definition, the documentation should be more specific about referring to it (and including a link would be preferred). If instead the documentation is incorrect, this language should be changed to describe the true meaning of \w.
¹ https://docs.python.org/3/library/re.html#index-32
² http://unicode.org/reports/tr18/
³ https://github.com/python/cpython/blob/master/Modules/_sre.c#L125
----------
assignee: docs@python
components: Documentation
messages: 355239
nosy: docs@python, snoopjedi
priority: normal
severity: normal
status: open
title: Description of '\w' behavior is vague in `re` documentation
_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38566>
_______________________________________
Josh Rosenberg <shadowranger+python@gmail.com> added the comment: The definition of \w, historically, has corresponded to the set of characters that can occur in legal variable names in C (alphanumeric ASCII plus underscores, making it equivalent to [a-zA-Z0-9_] for ASCII regex). That's why, on top of the definitely wordy alphabetic characters, and the arguably wordy numerics, it includes the underscore, _. That definition predates Unicode entirely, and Python is just building on it by expanding the definition of "alphanumeric" to encompass all alphanumeric characters in Unicode. We definitely can't remove underscores from the definition without breaking existing code which assumes a common subset of PCRE support (every regex flavor I know of includes underscores in \w). Adding the zero width characters seems of limited benefit (especially in the non-joiner case; if you're trying to pull out words, presumably you don't want to group letters across a non-joining boundary?). Basically, you're parsing "Unicode word characters" as "Unicode's definition of word characters", when it's really meant to mean "All word characters, not just ASCII". You omitted the clarifying remarks from the documentation though, the full description is:
Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore. If the ASCII flag is used, only [a-zA-Z0-9_] is matched.
That's about as precise as I think we can make it (because technically, some of the things that count as "word characters" aren't actually part of an "alphabet" in the technical definition). If you think there is a clearer way of expressing it, please suggest a better phrasing, and this can be fixed as a documentation bug. ---------- nosy: +josh.r _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue38566> _______________________________________
James Gerity <snoopjedi@gmail.com> added the comment: Cheers for the additional context. My recommendation would be to change the language to avoid confusion with the consortium's formal specifications. Describing what SRE does should be fine:
Matches any alphanumeric Unicode character, as well as '_'. If the ASCII flag is used, only [a-zA-Z0-9_] is matched.
I think it'd also be nice for the term "alphanumeric Unicode character" to link to the documentation for `str.isalnum()`, which provides enough clarity for the user to work out exactly what Unicode category properties will end up qualifying as a match. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue38566> _______________________________________
Change by Karthikeyan Singaravelan <tir.karthi@gmail.com>: ---------- nosy: +serhiy.storchaka type: -> behavior _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue38566> _______________________________________
participants (3)
-
James Gerity -
Josh Rosenberg
-
Karthikeyan Singaravelan