[issue24896] It is undocumented that re.UNICODE affects re.IGNORECASE
New submission from Leif Arne Storset: A non-ASCII string does not match a regular expression case-insensitively unless the UNICODE flag is set. This seems reasonable, but the documentation seems to imply that this is not the case. The example: import re # Does not match re.compile(u"неоднозначность", re.IGNORECASE) \ .findall(u"Неоднозначность") # Matches re.compile(u"неоднозначность", re.IGNORECASE | re.UNICODE) \ .findall(u"Неоднозначность") (In Python 3, it does not match if re.ASCII is given.) The documentation (2.7) says: re.UNICODE Make \w, \W, \b, \B, \d, \D, \s and \S dependent on the Unicode character properties database. (https://docs.python.org/2/library/re.html#re.UNICODE) My regex does not use any of those escapes, yet the regex changes behavior with the UNICODE flag. This leads to confusion when the regex doesn't match. The documentation is very specific about the behavior that changes with the flag, implying that behavior not mentioned is unaffected. Of course, it's easy to guess the correct (hopefully) solution. Still, I suggest changing the documentation to mention that re.IGNORECASE is affected. Looking at the source code, there seems to be further consequences (it mentions "Unicode locale") which may also warrant a mention. If you do want to avoid specifics, however, even a hand-wavy reference to something like "match according to Unicode" would help, because it implies that not only the escapes change behavior. In Python 3, there is a counterpart to the 2.7 problem: re.ASCII makes our Cyrillic string not match. Again, this behavior makes intuitive sense, but the documentation seems to indicate something different: re.ASCII Make \w, \W, \b, \B, \d, \D, \s and \S perform ASCII-only matching instead of full Unicode matching. This is only meaningful for Unicode patterns, and is ignored for byte patterns. … re.IGNORECASE Perform case-insensitive matching; expressions like [A-Z] will match lowercase letters, too. This is not affected by the current locale and works for Unicode characters as expected. re.ASCII does appear to affect re.IGNORECASE. Since this is the non-default case, however, I'm not sure it's worth calling it out. I'd be happy even if only the 2.7 docs change. ---------- assignee: docs@python components: Documentation messages: 248829 nosy: Leif Arne Storset, docs@python priority: normal severity: normal status: open title: It is undocumented that re.UNICODE affects re.IGNORECASE versions: Python 2.7, Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
R. David Murray added the comment: I think it would be reasonable to add re.IGNORECASE to the list of things affected, since it obviously does switch between using the unicode database and not doing so. ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Ezio Melotti <ezio.melotti@gmail.com>: ---------- components: +Regular Expressions nosy: +ezio.melotti, mrabarnett stage: -> needs patch type: -> enhancement versions: +Python 3.5, Python 3.6 -Python 3.4 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- versions: +Python 3.7 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Serhiy Storchaka added the comment: As was reported in issue30373 there is the same issue with re.LOCALE. This documentation issue should be easy for everyone who is fluent in English and familiar with the re module. ---------- keywords: +easy nosy: +serhiy.storchaka title: It is undocumented that re.UNICODE affects re.IGNORECASE -> It is undocumented that re.UNICODE and re.LOCALE affect re.IGNORECASE _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Jakub Wilk <jwilk@jwilk.net>: ---------- nosy: +jwilk _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Brian Ward <bri@aem7.net>: ---------- nosy: +Brian Ward _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Roundup Robot <devnull@psf.upfronthosting.co.za>: ---------- pull_requests: +1862 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Brian Ward <bri@aem7.net>: ---------- pull_requests: +1863 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Serhiy Storchaka added the comment: Actually the locale affects case-insensitively matching if use the re.LOCAL flag. The set of characters matched by b'[A-Z]' is locale-depending. For example in Turkish locale it can include the letters 'İ' and 'ı'. Only 8-bit locales are supported, not UTF-8 locales. In Unicode case-insensitive mode the expression '[A-Z]' matches not only Latin uppercase and lowercacase letters A-Z and a-z, but also characters 'İ', 'ı', 'ſ', and 'K'. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Brian Ward added the comment: OK, I'll look at this soon and come up with the next iteration. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Changes by Gregory P. Smith <greg@krypto.org>: ---------- pull_requests: +3340 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Serhiy Storchaka added the comment: This wording is not correct as I noted in msg294324. ---------- versions: -Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue24896> _______________________________________
Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment: I tried to correct the documentation in issue31714. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue24896> _______________________________________
Change by Serhiy Storchaka <storchaka+cpython@gmail.com>: ---------- resolution: -> fixed stage: needs patch -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue24896> _______________________________________
participants (8)
-
Brian Ward
-
Ezio Melotti
-
Gregory P. Smith
-
Jakub Wilk
-
Leif Arne Storset
-
R. David Murray
-
Roundup Robot
-
Serhiy Storchaka