[New-bugs-announce] [issue30838] re \w does not match some valid Unicode characters
David Lord
report at bugs.python.org
Mon Jul 3 11:39:29 EDT 2017
New submission from David Lord:
This came up while writing a regex to match characters that are valid in Python identifiers for Jinja. https://github.com/pallets/jinja/pull/731 `\w` matches all valid identifier characters except for 4 special cases:
import unicodedata
import re
import sys
cre = re.compile(r'\w')
for cp in range(sys.maxunicode + 1):
s = chr(cp)
if s.isidentifier() and not cre.match(s):
print(hex(cp), unicodedata.name(s))
0x1885 MONGOLIAN LETTER ALI GALI BALUDA
0x1886 MONGOLIAN LETTER ALI GALI THREE BALUDA
0x2118 SCRIPT CAPITAL P
0x212e ESTIMATED SYMBOL
Python < 3.6 matches the two Mongolian characters, not sure why 3.6 stopped matching them.
For our case, we just added them to a character set, `[\w\u1885\u1886\u2118\u212e]`.
It can cause unexpected behavior when using `\b`, since that's defined as the transition from `\w` to `\W` and those 4 characters aren't in `\w`. `re.match(r'\b[\w\u212e', '℮')` fails to match.
----------
components: Regular Expressions, Unicode
messages: 297603
nosy: davidism, ezio.melotti, haypo, mrabarnett
priority: normal
severity: normal
status: open
title: re \w does not match some valid Unicode characters
type: behavior
versions: Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30838>
_______________________________________
More information about the New-bugs-announce
mailing list