Mailman 3 March 2019 - Python-checkins

bpo-36150: Fix possible assertion failures due to _ctypes.c's PyCData_reduce(). (GH-12106) (GH-12642)
by Serhiy Storchaka March 31, 2019

March 31, 2019

https://github.com/python/cpython/commit/5e233951d931acc0e927100c51e9a27a27… commit: 5e233951d931acc0e927100c51e9a27a2791b6a5 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2019-03-31T20:15:11+03:00 summary: bpo-36150: Fix possible assertion failures due to _ctypes.c's PyCData_reduce(). (GH-12106) (GH-12642) (cherry picked from commit 5f2c50810a67982b0c80f6d3258fee3647f67005) Co-authored-by: Zackery Spytz <zspytz(a)gmail.com> files: M Modules/_ctypes/_ctypes.c diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 48ad696e1078..be0b321bad03 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -2663,10 +2663,11 @@ PyCData_reduce(PyObject *myself, PyObject *args) "ctypes objects containing pointers cannot be pickled"); return NULL; } - return Py_BuildValue("O(O(NN))", - _unpickle, - Py_TYPE(myself), - PyObject_GetAttrString(myself, "__dict__"), + PyObject *dict = PyObject_GetAttrString(myself, "__dict__"); + if (dict == NULL) { + return NULL; + } + return Py_BuildValue("O(O(NN))", _unpickle, Py_TYPE(myself), dict, PyBytes_FromStringAndSize(self->b_ptr, self->b_size)); }

1 0

bpo-36150: Fix possible assertion failures due to _ctypes.c's PyCData_reduce(). (GH-12106) (GH-12643)
by Serhiy Storchaka March 31, 2019

March 31, 2019

https://github.com/python/cpython/commit/a110817c080ca3c2f3262350b5d7e0c058… commit: a110817c080ca3c2f3262350b5d7e0c0582527e6 branch: 2.7 author: Zackery Spytz <zspytz(a)gmail.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2019-03-31T20:14:16+03:00 summary: bpo-36150: Fix possible assertion failures due to _ctypes.c's PyCData_reduce(). (GH-12106) (GH-12643) (cherry picked from commit 5f2c50810a67982b0c80f6d3258fee3647f67005) files: M Modules/_ctypes/_ctypes.c diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 3a3aabbb6cb5..33e224386fb1 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -2795,16 +2795,18 @@ static PyObject * PyCData_reduce(PyObject *_self, PyObject *args) { CDataObject *self = (CDataObject *)_self; + PyObject *dict; if (PyObject_stgdict(_self)->flags & (TYPEFLAG_ISPOINTER|TYPEFLAG_HASPOINTER)) { PyErr_SetString(PyExc_ValueError, "ctypes objects containing pointers cannot be pickled"); return NULL; } - return Py_BuildValue("O(O(NN))", - _unpickle, - Py_TYPE(_self), - PyObject_GetAttrString(_self, "__dict__"), + dict = PyObject_GetAttrString(_self, "__dict__"); + if (dict == NULL) { + return NULL; + } + return Py_BuildValue("O(O(NN))", _unpickle, Py_TYPE(_self), dict, PyString_FromStringAndSize(self->b_ptr, self->b_size)); }

1 0

bpo-36150: Fix possible assertion failures due to _ctypes.c's PyCData_reduce(). (GH-12106)
by Serhiy Storchaka March 31, 2019

March 31, 2019

https://github.com/python/cpython/commit/5f2c50810a67982b0c80f6d3258fee3647… commit: 5f2c50810a67982b0c80f6d3258fee3647f67005 branch: master author: Zackery Spytz <zspytz(a)gmail.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2019-03-31T19:02:11+03:00 summary: bpo-36150: Fix possible assertion failures due to _ctypes.c's PyCData_reduce(). (GH-12106) files: M Modules/_ctypes/_ctypes.c diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 03f8e756092c..b3a20309472d 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -2743,10 +2743,11 @@ PyCData_reduce(PyObject *myself, PyObject *args) "ctypes objects containing pointers cannot be pickled"); return NULL; } - return Py_BuildValue("O(O(NN))", - _unpickle, - Py_TYPE(myself), - PyObject_GetAttrString(myself, "__dict__"), + PyObject *dict = PyObject_GetAttrString(myself, "__dict__"); + if (dict == NULL) { + return NULL; + } + return Py_BuildValue("O(O(NN))", _unpickle, Py_TYPE(myself), dict, PyBytes_FromStringAndSize(self->b_ptr, self->b_size)); }

1 0

bpo-35947: Fix a compiler warning in _ctypes.c's StructUnionType_paramfunc(). (GH-12629)
by Serhiy Storchaka March 31, 2019

March 31, 2019

https://github.com/python/cpython/commit/48600c72c1afe1096c2412a135a43f8cdd… commit: 48600c72c1afe1096c2412a135a43f8cdd895195 branch: master author: Zackery Spytz <zspytz(a)gmail.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2019-03-31T19:00:12+03:00 summary: bpo-35947: Fix a compiler warning in _ctypes.c's StructUnionType_paramfunc(). (GH-12629) files: M Modules/_ctypes/_ctypes.c diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 7479edaf1110..03f8e756092c 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -406,7 +406,7 @@ StructUnionType_paramfunc(CDataObject *self) CDataObject *copied_self; StgDictObject *stgdict; - if (self->b_size > sizeof(void*)) { + if ((size_t)self->b_size > sizeof(void*)) { void *new_ptr = PyMem_Malloc(self->b_size); if (new_ptr == NULL) return NULL;

1 0

bpo-36085: Add installer check for KB2533625 (GH-12636)
by Steve Dower March 31, 2019

March 31, 2019

https://github.com/python/cpython/commit/79da388a4016e24c4258dcc62cd0fa9dde… commit: 79da388a4016e24c4258dcc62cd0fa9dde0acb5b branch: master author: Steve Dower <steve.dower(a)microsoft.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T20:58:17-07:00 summary: bpo-36085: Add installer check for KB2533625 (GH-12636) files: M Doc/whatsnew/3.8.rst M Tools/msi/build.bat M Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp M Tools/msi/lib/lib_files.wxs diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index f0423c376fcd..ccd8bbd81042 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -754,7 +754,9 @@ Changes in the Python API used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs - directory while loading your library. + directory while loading your library. Note that Windows 7 users will need to + ensure that Windows Update KB2533625 has been installed (this is also verified + by the installer). (See :issue:`36085`.) diff --git a/Tools/msi/build.bat b/Tools/msi/build.bat index 8fa612e9ddc0..532cebc5b511 100644 --- a/Tools/msi/build.bat +++ b/Tools/msi/build.bat @@ -6,7 +6,7 @@ set PCBUILD=%D%..\..\PCbuild\ set BUILDX86= set BUILDX64= set BUILDDOC= -set BUILDTEST=--test-marker +set BUILDTEST= set BUILDPACK= set REBUILD= @@ -16,6 +16,7 @@ if "%~1" EQU "-x86" (set BUILDX86=1) && shift && goto CheckOpts if "%~1" EQU "-x64" (set BUILDX64=1) && shift && goto CheckOpts if "%~1" EQU "--doc" (set BUILDDOC=1) && shift && goto CheckOpts if "%~1" EQU "--no-test-marker" (set BUILDTEST=) && shift && goto CheckOpts +if "%~1" EQU "--test-marker" (set BUILDTEST=--test-marker) && shift && goto CheckOpts if "%~1" EQU "--pack" (set BUILDPACK=1) && shift && goto CheckOpts if "%~1" EQU "-r" (set REBUILD=-r) && shift && goto CheckOpts @@ -69,11 +70,12 @@ if defined BUILDX64 ( exit /B 0 :Help -echo build.bat [-x86] [-x64] [--doc] [-h] [--no-test-marker] [--pack] [-r] +echo build.bat [-x86] [-x64] [--doc] [-h] [--test-marker] [--pack] [-r] echo. echo -x86 Build x86 installers echo -x64 Build x64 installers echo --doc Build CHM documentation -echo --no-test-marker Build without test markers +echo --test-marker Build with test markers +echo --no-test-marker Build without test markers (default) echo --pack Embed core MSIs into installer echo -r Rebuild rather than incremental build diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp index 2e468b7e57b2..7cd8fb8e0583 100644 --- a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp +++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp @@ -2989,9 +2989,20 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { LOC_STRING *pLocString = nullptr; if (IsWindowsServer()) { - if (IsWindowsVersionOrGreater(6, 1, 1)) { - BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Target OS is Windows Server 2008 R2 or later"); + if (IsWindowsVersionOrGreater(6, 2, 0)) { + BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Target OS is Windows Server 2012 or later"); return; + } else if (IsWindowsVersionOrGreater(6, 1, 1)) { + HMODULE hKernel32 = GetModuleHandleW(L"kernel32"); + if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) { + BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533625"); + BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue."); + /* The "MissingSP1" error also specifies updates are required */ + LocGetString(_wixLoc, L"#(loc.FailureWS2K8R2MissingSP1)", &pLocString); + } else { + BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Target OS is Windows Server 2008 R2 or later"); + return; + } } else if (IsWindowsVersionOrGreater(6, 1, 0)) { BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2"); BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Service Pack 1 is required to continue installation"); @@ -3009,9 +3020,20 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication { LocGetString(_wixLoc, L"#(loc.FailureWS2K3OrEarlier)", &pLocString); } } else { - if (IsWindows7SP1OrGreater()) { - BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Target OS is Windows 7 SP1 or later"); + if (IsWindows8OrGreater()) { + BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Target OS is Windows 8 or later"); return; + } else if (IsWindows7SP1OrGreater()) { + HMODULE hKernel32 = GetModuleHandleW(L"kernel32"); + if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) { + BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533625"); + BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue."); + /* The "MissingSP1" error also specifies updates are required */ + LocGetString(_wixLoc, L"#(loc.FailureWin7MissingSP1)", &pLocString); + } else { + BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Target OS is Windows 7 SP1 or later"); + return; + } } else if (IsWindows7OrGreater()) { BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 RTM"); BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Service Pack 1 is required to continue installation"); diff --git a/Tools/msi/lib/lib_files.wxs b/Tools/msi/lib/lib_files.wxs index 472bacf20dba..b462372512f6 100644 --- a/Tools/msi/lib/lib_files.wxs +++ b/Tools/msi/lib/lib_files.wxs @@ -62,9 +62,6 @@ <Component Id="libssl.pdb" Directory="DLLs" Guid="*"> <File Name="libssl$(var.ssltag).pdb" KeyPath="yes" /> </Component> - <Component Id="libffi.pdb" Directory="DLLs" Guid="*"> - <File Name="libffi-7.pdb" KeyPath="yes" /> - </Component> <Component Id="venvlauncher.pdb" Directory="Lib_venv_scripts_nt__pdbs" Guid="*"> <File Name="python.pdb" Source="venvlauncher.pdb" KeyPath="yes" /> </Component>

1 0

bpo-36085: Add additional load flag to ensure DLL loads successfully (GH-12633)
by Steve Dower March 31, 2019

March 31, 2019

https://github.com/python/cpython/commit/ac19d9652799412404aef6b357a01057df… commit: ac19d9652799412404aef6b357a01057df34e005 branch: master author: Steve Dower <steve.dower(a)microsoft.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T17:14:46-07:00 summary: bpo-36085: Add additional load flag to ensure DLL loads successfully (GH-12633) files: M Lib/ctypes/test/test_loading.py diff --git a/Lib/ctypes/test/test_loading.py b/Lib/ctypes/test/test_loading.py index be367c6fa352..9b97d8008604 100644 --- a/Lib/ctypes/test/test_loading.py +++ b/Lib/ctypes/test/test_loading.py @@ -167,7 +167,8 @@ def should_fail(command): # Full path load with DLL_LOAD_DIR should succeed should_pass("WinDLL(nt._getfullpathname('_sqlite3.dll'), " + - "winmode=nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR)") + "winmode=nt._LOAD_LIBRARY_SEARCH_SYSTEM32|" + + "nt._LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR)") # User-specified directory should succeed should_pass("import os; p = os.add_dll_directory(os.getcwd());" +

1 0

bpo-36010: Add venv to the nuget distribution (GH-12367)
by Steve Dower March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/3e78c7c30553baf72b7eb6fe3384d88fff… commit: 3e78c7c30553baf72b7eb6fe3384d88fff549906 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Steve Dower <steve.dower(a)microsoft.com> date: 2019-03-30T14:47:12-07:00 summary: bpo-36010: Add venv to the nuget distribution (GH-12367) (cherry picked from commit e724152796a5a41544f52054506c6c2248242a5d) Co-authored-by: Paul Moore <p.f.moore(a)gmail.com> files: A Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst M PC/layout/support/options.py M Tools/nuget/make_pkg.proj diff --git a/Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst b/Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst new file mode 100644 index 000000000000..32c57c49ae41 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst @@ -0,0 +1 @@ +Add the venv standard library module to the nuget distribution for Windows. \ No newline at end of file diff --git a/PC/layout/support/options.py b/PC/layout/support/options.py index 76d9e34e1f46..22492f220d60 100644 --- a/PC/layout/support/options.py +++ b/PC/layout/support/options.py @@ -53,7 +53,15 @@ def public(f): }, "nuget": { "help": "nuget package", - "options": ["stable", "pip", "distutils", "dev", "props"], + "options": [ + "dev", + "tools", + "pip", + "stable", + "distutils", + "venv", + "props" + ], }, "default": { "help": "development kit package", diff --git a/Tools/nuget/make_pkg.proj b/Tools/nuget/make_pkg.proj index e093a6d0bd76..5638952ac9f9 100644 --- a/Tools/nuget/make_pkg.proj +++ b/Tools/nuget/make_pkg.proj @@ -28,7 +28,7 @@ <PythonArguments>$(PythonArguments) -b "$(BuildPath.TrimEnd(`\`))" -s "$(PySourcePath.TrimEnd(`\`))"</PythonArguments> <PythonArguments>$(PythonArguments) -t "$(IntermediateOutputPath)obj"</PythonArguments> <PythonArguments>$(PythonArguments) --copy "$(IntermediateOutputPath)pkg"</PythonArguments> - <PythonArguments>$(PythonArguments) --include-dev --include-tools --include-pip --include-stable --include-launcher --include-props</PythonArguments> + <PythonArguments>$(PythonArguments) --preset-nuget</PythonArguments> <PackageArguments Condition="$(Packages) != ''">"$(IntermediateOutputPath)pkg\pip.exe" -B -m pip install -U $(Packages)</PackageArguments>

1 0

bpo-36010: Add venv to the nuget distribution (GH-12367)
by Steve Dower March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/e724152796a5a41544f52054506c6c2248… commit: e724152796a5a41544f52054506c6c2248242a5d branch: master author: Paul Moore <p.f.moore(a)gmail.com> committer: Steve Dower <steve.dower(a)microsoft.com> date: 2019-03-30T09:32:05-07:00 summary: bpo-36010: Add venv to the nuget distribution (GH-12367) files: A Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst M PC/layout/support/options.py M Tools/nuget/make_pkg.proj diff --git a/Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst b/Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst new file mode 100644 index 000000000000..32c57c49ae41 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2019-03-16-10-24-58.bpo-36010.dttWfp.rst @@ -0,0 +1 @@ +Add the venv standard library module to the nuget distribution for Windows. \ No newline at end of file diff --git a/PC/layout/support/options.py b/PC/layout/support/options.py index 76d9e34e1f46..22492f220d60 100644 --- a/PC/layout/support/options.py +++ b/PC/layout/support/options.py @@ -53,7 +53,15 @@ def public(f): }, "nuget": { "help": "nuget package", - "options": ["stable", "pip", "distutils", "dev", "props"], + "options": [ + "dev", + "tools", + "pip", + "stable", + "distutils", + "venv", + "props" + ], }, "default": { "help": "development kit package", diff --git a/Tools/nuget/make_pkg.proj b/Tools/nuget/make_pkg.proj index e093a6d0bd76..5638952ac9f9 100644 --- a/Tools/nuget/make_pkg.proj +++ b/Tools/nuget/make_pkg.proj @@ -28,7 +28,7 @@ <PythonArguments>$(PythonArguments) -b "$(BuildPath.TrimEnd(`\`))" -s "$(PySourcePath.TrimEnd(`\`))"</PythonArguments> <PythonArguments>$(PythonArguments) -t "$(IntermediateOutputPath)obj"</PythonArguments> <PythonArguments>$(PythonArguments) --copy "$(IntermediateOutputPath)pkg"</PythonArguments> - <PythonArguments>$(PythonArguments) --include-dev --include-tools --include-pip --include-stable --include-launcher --include-props</PythonArguments> + <PythonArguments>$(PythonArguments) --preset-nuget</PythonArguments> <PackageArguments Condition="$(Packages) != ''">"$(IntermediateOutputPath)pkg\pip.exe" -B -m pip install -U $(Packages)</PackageArguments>

1 0

bpo-36384: Remove check for leading zeroes in IPv4 addresses (GH-12577)
by Nick Coghlan March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/e653d4d8e820a7a004ad399530af0135b4… commit: e653d4d8e820a7a004ad399530af0135b45db27a branch: master author: Joel Croteau <jcroteau(a)gmail.com> committer: Nick Coghlan <ncoghlan(a)gmail.com> date: 2019-03-31T00:53:48+10:00 summary: bpo-36384: Remove check for leading zeroes in IPv4 addresses (GH-12577) Stop rejecting IPv4 octets with leading zeroes as ambiguously octal. Plenty of other tools generate decimal IPv4 octets with leading zeroes, so keeping this check hurts interoperability. Patch by Joel Croteau. files: A Misc/NEWS.d/next/Library/2019-03-27-02-09-22.bpo-36385.we2F45.rst M Lib/ipaddress.py M Lib/test/test_ipaddress.py diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index 15507d61dec8..a88cf3d0b7c5 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1165,12 +1165,6 @@ def _parse_octet(cls, octet_str): raise ValueError(msg % octet_str) # Convert to integer (we know digits are legal) octet_int = int(octet_str, 10) - # Any octets that look like they *might* be written in octal, - # and which don't look exactly the same in both octal and - # decimal are rejected as ambiguous - if octet_int > 7 and octet_str[0] == '0': - msg = "Ambiguous (octal/decimal) value in %r not permitted" - raise ValueError(msg % octet_str) if octet_int > 255: raise ValueError("Octet %d (> 255) not permitted" % octet_int) return octet_int diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py index 0e0753f34c49..53f6f128443c 100644 --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -92,11 +92,14 @@ def pickle_test(self, addr): y = pickle.loads(pickle.dumps(x, proto)) self.assertEqual(y, x) + class CommonTestMixin_v4(CommonTestMixin): def test_leading_zeros(self): self.assertInstancesEqual("000.000.000.000", "0.0.0.0") self.assertInstancesEqual("192.168.000.001", "192.168.0.1") + self.assertInstancesEqual("016.016.016.016", "16.16.16.16") + self.assertInstancesEqual("001.000.008.016", "1.0.8.16") def test_int(self): self.assertInstancesEqual(0, "0.0.0.0") @@ -229,15 +232,6 @@ def assertBadOctet(addr, octet): assertBadOctet("1.2.3.4::", "4::") assertBadOctet("1.a.2.3", "a") - def test_octal_decimal_ambiguity(self): - def assertBadOctet(addr, octet): - msg = "Ambiguous (octal/decimal) value in %r not permitted in %r" - with self.assertAddressError(re.escape(msg % (octet, addr))): - ipaddress.IPv4Address(addr) - - assertBadOctet("016.016.016.016", "016") - assertBadOctet("001.000.008.016", "008") - def test_octet_length(self): def assertBadOctet(addr, octet): msg = "At most 3 characters permitted in %r in %r" diff --git a/Misc/NEWS.d/next/Library/2019-03-27-02-09-22.bpo-36385.we2F45.rst b/Misc/NEWS.d/next/Library/2019-03-27-02-09-22.bpo-36385.we2F45.rst new file mode 100644 index 000000000000..26f6dd7d5f47 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-03-27-02-09-22.bpo-36385.we2F45.rst @@ -0,0 +1 @@ +Stop rejecting IPv4 octets for being ambiguously octal. Leading zeros are ignored, and no longer are assumed to specify octal octets. Octets are always decimal numbers. Octets must still be no more than three digits, including leading zeroes.

1 0

bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603) (GH-12627)
by Serhiy Storchaka March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/bd48280cb66544827952ca91e326cbb178… commit: bd48280cb66544827952ca91e326cbb178c8c461 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2019-03-30T15:52:41+02:00 summary: bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603) (GH-12627) The bug occurred when the encoded surrogate character is passed to the incremental decoder in two chunks. (cherry picked from commit 7a465cb5ee7e298cae626ace1fc3e7d97df79f2e) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2019-03-28-15-22-45.bpo-24214.tZ6lYU.rst M Lib/test/test_codecs.py M Objects/unicodeobject.c diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 293dfbc61aba..5ba2c7bdc5f8 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -401,6 +401,15 @@ def test_lone_surrogates(self): self.assertEqual(test_sequence.decode(self.encoding, "backslashreplace"), before + backslashreplace + after) + def test_incremental_surrogatepass(self): + # Test incremental decoder for surrogatepass handler: + # see issue #24214 + data = '\uD901'.encode(self.encoding, 'surrogatepass') + for i in range(1, len(data)): + dec = codecs.getincrementaldecoder(self.encoding)('surrogatepass') + self.assertEqual(dec.decode(data[:i]), '') + self.assertEqual(dec.decode(data[i:], True), '\uD901') + class UTF32Test(ReadTest, unittest.TestCase): encoding = "utf-32" diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-03-28-15-22-45.bpo-24214.tZ6lYU.rst b/Misc/NEWS.d/next/Core and Builtins/2019-03-28-15-22-45.bpo-24214.tZ6lYU.rst new file mode 100644 index 000000000000..abb27591f78a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-03-28-15-22-45.bpo-24214.tZ6lYU.rst @@ -0,0 +1,2 @@ +Fixed support of the surrogatepass error handler in the UTF-8 incremental +decoder. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index adcf69d4e539..e18937981bb7 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4890,6 +4890,9 @@ PyUnicode_DecodeUTF8Stateful(const char *s, case 2: case 3: case 4: + if (s == end || consumed) { + goto End; + } errmsg = "invalid continuation byte"; startinpos = s - starts; endinpos = startinpos + ch - 1;

1 0