[Python-checkins] bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958)

[Miss Islington (bot)]

https://github.com/python/cpython/commit/0e2b76ea4e48d0fc1ca34ae4ffbb2fd6c19664bb
commit: 0e2b76ea4e48d0fc1ca34ae4ffbb2fd6c19664bb
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2018-06-15T05:21:55-07:00
summary:

bpo-29456: Fix bugs in unicodedata.normalize: u1176, u11a7 and u11c3 (GH-1958)


Hangul composition check boundaries are wrong for the second character
([0x1161, 0x1176) instead of [0x1161, 0x1176]) and third character ((0x11A7, 0x11C3)
instead of [0x11A7, 0x11C3]).
(cherry picked from commit d134809cd3764c6a634eab7bb8995e3e2eff14d5)

Co-authored-by: Wonsup Yoon <pusnow at me.com>

files:
A Misc/NEWS.d/next/Library/2017-08-24-17-55-39.bpo-29456.XaB3MP.rst
M Lib/test/test_unicodedata.py
M Misc/ACKS
M Modules/unicodedata.c

diff --git a/Lib/test/test_unicodedata.py b/Lib/test/test_unicodedata.py
index 99dd0dec9d11..170778fa977d 100644
--- a/Lib/test/test_unicodedata.py
+++ b/Lib/test/test_unicodedata.py
@@ -208,6 +208,19 @@ def test_issue10254(self):
         b = 'C\u0338' * 20  + '\xC7'
         self.assertEqual(self.db.normalize('NFC', a), b)
 
+    def test_issue29456(self):
+        # Fix #29456
+        u1176_str_a = '\u1100\u1176\u11a8'
+        u1176_str_b = '\u1100\u1176\u11a8'
+        u11a7_str_a = '\u1100\u1175\u11a7'
+        u11a7_str_b = '\uae30\u11a7'
+        u11c3_str_a = '\u1100\u1175\u11c3'
+        u11c3_str_b = '\uae30\u11c3'
+        self.assertEqual(self.db.normalize('NFC', u1176_str_a), u1176_str_b)
+        self.assertEqual(self.db.normalize('NFC', u11a7_str_a), u11a7_str_b)
+        self.assertEqual(self.db.normalize('NFC', u11c3_str_a), u11c3_str_b)
+
+
     def test_east_asian_width(self):
         eaw = self.db.east_asian_width
         self.assertRaises(TypeError, eaw, b'a')
diff --git a/Misc/ACKS b/Misc/ACKS
index 7234c0a766fd..b1e4ec68e35c 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1791,6 +1791,7 @@ Jason Yeo
 EungJun Yi
 Bob Yodlowski
 Danny Yoo
+Wonsup Yoon
 Rory Yorke
 George Yoshida
 Kazuhiro Yoshida
diff --git a/Misc/NEWS.d/next/Library/2017-08-24-17-55-39.bpo-29456.XaB3MP.rst b/Misc/NEWS.d/next/Library/2017-08-24-17-55-39.bpo-29456.XaB3MP.rst
new file mode 100644
index 000000000000..9b30bf654bd0
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-08-24-17-55-39.bpo-29456.XaB3MP.rst
@@ -0,0 +1 @@
+Fix bugs in hangul normalization: u1176, u11a7 and u11c3
diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
index 7a9a964a0f6c..e8788f5036dd 100644
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -681,15 +681,19 @@ nfc_nfkc(PyObject *self, PyObject *input, int k)
       if (LBase <= code && code < (LBase+LCount) &&
           i + 1 < len &&
           VBase <= PyUnicode_READ(kind, data, i+1) &&
-          PyUnicode_READ(kind, data, i+1) <= (VBase+VCount)) {
+          PyUnicode_READ(kind, data, i+1) < (VBase+VCount)) {
+          /* check L character is a modern leading consonant (0x1100 ~ 0x1112)
+             and V character is a modern vowel (0x1161 ~ 0x1175). */
           int LIndex, VIndex;
           LIndex = code - LBase;
           VIndex = PyUnicode_READ(kind, data, i+1) - VBase;
           code = SBase + (LIndex*VCount+VIndex)*TCount;
           i+=2;
           if (i < len &&
-              TBase <= PyUnicode_READ(kind, data, i) &&
-              PyUnicode_READ(kind, data, i) <= (TBase+TCount)) {
+              TBase < PyUnicode_READ(kind, data, i) &&
+              PyUnicode_READ(kind, data, i) < (TBase+TCount)) {
+              /* check T character is a modern trailing consonant
+                 (0x11A8 ~ 0x11C2). */
               code += PyUnicode_READ(kind, data, i)-TBase;
               i++;
           }