From webhook-mailer at python.org Sat Sep 1 04:57:02 2018 From: webhook-mailer at python.org (Ivan Levkivskyi) Date: Sat, 01 Sep 2018 08:57:02 -0000 Subject: [Python-checkins] Fix typo in typing.py module docstring (#9014) Message-ID: https://github.com/python/cpython/commit/5265b3a98b376684e361b62d0728483b26f493f2 commit: 5265b3a98b376684e361b62d0728483b26f493f2 branch: master author: Tim McNamara committer: Ivan Levkivskyi date: 2018-09-01T09:56:58+01:00 summary: Fix typo in typing.py module docstring (#9014) "explicitelly" ? "explicitly" files: M Lib/typing.py diff --git a/Lib/typing.py b/Lib/typing.py index 48992633ac34..445a42492b6b 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -2,7 +2,7 @@ The typing module: Support for gradual typing as defined by PEP 484. At large scale, the structure of the module is following: -* Imports and exports, all public names should be explicitelly added to __all__. +* Imports and exports, all public names should be explicitly added to __all__. * Internal helper functions: these should never be used in code outside this module. * _SpecialForm and its instances (special forms): Any, NoReturn, ClassVar, Union, Optional * Two classes whose instances can be type arguments in addition to types: ForwardRef and TypeVar From solipsis at pitrou.net Sat Sep 1 05:10:43 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 01 Sep 2018 09:10:43 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-4 Message-ID: <20180901091043.1.CE09C2F9448AA2BC@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [-7, 1, 0] memory blocks, sum=-6 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_fork leaked [0, 0, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogxJiVO_', '--timeout', '7200'] From webhook-mailer at python.org Sat Sep 1 05:15:45 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 01 Sep 2018 09:15:45 -0000 Subject: [Python-checkins] Fix typo in typing.py module docstring (GH-9014) Message-ID: https://github.com/python/cpython/commit/f6d70b8b1708382f09b008bd1cd8792e00b82afb commit: f6d70b8b1708382f09b008bd1cd8792e00b82afb branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-01T05:15:40-04:00 summary: Fix typo in typing.py module docstring (GH-9014) "explicitelly" ? "explicitly" (cherry picked from commit 5265b3a98b376684e361b62d0728483b26f493f2) Co-authored-by: Tim McNamara files: M Lib/typing.py diff --git a/Lib/typing.py b/Lib/typing.py index 48992633ac34..445a42492b6b 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -2,7 +2,7 @@ The typing module: Support for gradual typing as defined by PEP 484. At large scale, the structure of the module is following: -* Imports and exports, all public names should be explicitelly added to __all__. +* Imports and exports, all public names should be explicitly added to __all__. * Internal helper functions: these should never be used in code outside this module. * _SpecialForm and its instances (special forms): Any, NoReturn, ClassVar, Union, Optional * Two classes whose instances can be type arguments in addition to types: ForwardRef and TypeVar From webhook-mailer at python.org Sat Sep 1 18:14:01 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Sat, 01 Sep 2018 22:14:01 -0000 Subject: [Python-checkins] closes bpo-34555: Fix incorrectly nested test for HAVE_LINUX_VM_SOCKETS_H (GH-9016) Message-ID: https://github.com/python/cpython/commit/2d7102e726e973ab2d307aa9748c7ec433677877 commit: 2d7102e726e973ab2d307aa9748c7ec433677877 branch: master author: Thomas Herzog committer: Benjamin Peterson date: 2018-09-01T15:13:57-07:00 summary: closes bpo-34555: Fix incorrectly nested test for HAVE_LINUX_VM_SOCKETS_H (GH-9016) files: A Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst M Modules/socketmodule.h diff --git a/Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst b/Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst new file mode 100644 index 000000000000..7e61c4fb20ab --- /dev/null +++ b/Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst @@ -0,0 +1,2 @@ +Fix for case where it was not possible to have both +``HAVE_LINUX_VM_SOCKETS_H`` and ``HAVE_SOCKADDR_ALG`` be undefined. diff --git a/Modules/socketmodule.h b/Modules/socketmodule.h index bce74c3da8fd..0b2edc158782 100644 --- a/Modules/socketmodule.h +++ b/Modules/socketmodule.h @@ -101,39 +101,40 @@ typedef int socklen_t; #include #endif -#ifdef HAVE_SOCKADDR_ALG -#include -#ifndef AF_ALG -#define AF_ALG 38 -#endif -#ifndef SOL_ALG -#define SOL_ALG 279 -#endif - #ifdef HAVE_LINUX_VM_SOCKETS_H # include #else # undef AF_VSOCK #endif +#ifdef HAVE_SOCKADDR_ALG + +# include +# ifndef AF_ALG +# define AF_ALG 38 +# endif +# ifndef SOL_ALG +# define SOL_ALG 279 +# endif + /* Linux 3.19 */ -#ifndef ALG_SET_AEAD_ASSOCLEN -#define ALG_SET_AEAD_ASSOCLEN 4 -#endif -#ifndef ALG_SET_AEAD_AUTHSIZE -#define ALG_SET_AEAD_AUTHSIZE 5 -#endif +# ifndef ALG_SET_AEAD_ASSOCLEN +# define ALG_SET_AEAD_ASSOCLEN 4 +# endif +# ifndef ALG_SET_AEAD_AUTHSIZE +# define ALG_SET_AEAD_AUTHSIZE 5 +# endif /* Linux 4.8 */ -#ifndef ALG_SET_PUBKEY -#define ALG_SET_PUBKEY 6 -#endif +# ifndef ALG_SET_PUBKEY +# define ALG_SET_PUBKEY 6 +# endif -#ifndef ALG_OP_SIGN -#define ALG_OP_SIGN 2 -#endif -#ifndef ALG_OP_VERIFY -#define ALG_OP_VERIFY 3 -#endif +# ifndef ALG_OP_SIGN +# define ALG_OP_SIGN 2 +# endif +# ifndef ALG_OP_VERIFY +# define ALG_OP_VERIFY 3 +# endif #endif /* HAVE_SOCKADDR_ALG */ From webhook-mailer at python.org Sat Sep 1 18:30:49 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 01 Sep 2018 22:30:49 -0000 Subject: [Python-checkins] closes bpo-34555: Fix incorrectly nested test for HAVE_LINUX_VM_SOCKETS_H (GH-9016) Message-ID: https://github.com/python/cpython/commit/4c532da1209bd20ba07f18448134f32ace8c54f7 commit: 4c532da1209bd20ba07f18448134f32ace8c54f7 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-01T18:30:44-04:00 summary: closes bpo-34555: Fix incorrectly nested test for HAVE_LINUX_VM_SOCKETS_H (GH-9016) (cherry picked from commit 2d7102e726e973ab2d307aa9748c7ec433677877) Co-authored-by: Thomas Herzog files: A Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst M Modules/socketmodule.h diff --git a/Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst b/Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst new file mode 100644 index 000000000000..7e61c4fb20ab --- /dev/null +++ b/Misc/NEWS.d/next/Build/2018-08-31-19-41-09.bpo-34555.dfQcnm.rst @@ -0,0 +1,2 @@ +Fix for case where it was not possible to have both +``HAVE_LINUX_VM_SOCKETS_H`` and ``HAVE_SOCKADDR_ALG`` be undefined. diff --git a/Modules/socketmodule.h b/Modules/socketmodule.h index bce74c3da8fd..0b2edc158782 100644 --- a/Modules/socketmodule.h +++ b/Modules/socketmodule.h @@ -101,39 +101,40 @@ typedef int socklen_t; #include #endif -#ifdef HAVE_SOCKADDR_ALG -#include -#ifndef AF_ALG -#define AF_ALG 38 -#endif -#ifndef SOL_ALG -#define SOL_ALG 279 -#endif - #ifdef HAVE_LINUX_VM_SOCKETS_H # include #else # undef AF_VSOCK #endif +#ifdef HAVE_SOCKADDR_ALG + +# include +# ifndef AF_ALG +# define AF_ALG 38 +# endif +# ifndef SOL_ALG +# define SOL_ALG 279 +# endif + /* Linux 3.19 */ -#ifndef ALG_SET_AEAD_ASSOCLEN -#define ALG_SET_AEAD_ASSOCLEN 4 -#endif -#ifndef ALG_SET_AEAD_AUTHSIZE -#define ALG_SET_AEAD_AUTHSIZE 5 -#endif +# ifndef ALG_SET_AEAD_ASSOCLEN +# define ALG_SET_AEAD_ASSOCLEN 4 +# endif +# ifndef ALG_SET_AEAD_AUTHSIZE +# define ALG_SET_AEAD_AUTHSIZE 5 +# endif /* Linux 4.8 */ -#ifndef ALG_SET_PUBKEY -#define ALG_SET_PUBKEY 6 -#endif +# ifndef ALG_SET_PUBKEY +# define ALG_SET_PUBKEY 6 +# endif -#ifndef ALG_OP_SIGN -#define ALG_OP_SIGN 2 -#endif -#ifndef ALG_OP_VERIFY -#define ALG_OP_VERIFY 3 -#endif +# ifndef ALG_OP_SIGN +# define ALG_OP_SIGN 2 +# endif +# ifndef ALG_OP_VERIFY +# define ALG_OP_VERIFY 3 +# endif #endif /* HAVE_SOCKADDR_ALG */ From webhook-mailer at python.org Sat Sep 1 21:59:30 2018 From: webhook-mailer at python.org (Zachary Ware) Date: Sun, 02 Sep 2018 01:59:30 -0000 Subject: [Python-checkins] Fix struct sequence glossary entry grammar (GH-9030) Message-ID: https://github.com/python/cpython/commit/98b976a2f82ba5f50cf6846338f644ca6c64f47d commit: 98b976a2f82ba5f50cf6846338f644ca6c64f47d branch: master author: Zachary Ware committer: GitHub date: 2018-09-01T20:59:27-05:00 summary: Fix struct sequence glossary entry grammar (GH-9030) ... by removing a superfluous "either". Reported by ?????? ???????? on docs at . files: M Doc/glossary.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 69960579f2c0..16fc7f0783c8 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -1012,7 +1012,7 @@ Glossary struct sequence A tuple with named elements. Struct sequences expose an interface similar - to :term:`named tuple` in that elements can either be accessed either by + to :term:`named tuple` in that elements can be accessed either by index or as an attribute. However, they do not have any of the named tuple methods like :meth:`~collections.somenamedtuple._make` or :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences From webhook-mailer at python.org Sat Sep 1 22:13:38 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 02 Sep 2018 02:13:38 -0000 Subject: [Python-checkins] Fix struct sequence glossary entry grammar (GH-9030) Message-ID: https://github.com/python/cpython/commit/f4c865eda8f57dedaeaa16530ab6f1024bfb94de commit: f4c865eda8f57dedaeaa16530ab6f1024bfb94de branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-01T22:13:34-04:00 summary: Fix struct sequence glossary entry grammar (GH-9030) ... by removing a superfluous "either". Reported by ?????? ???????? on docs at . (cherry picked from commit 98b976a2f82ba5f50cf6846338f644ca6c64f47d) Co-authored-by: Zachary Ware files: M Doc/glossary.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 69960579f2c0..16fc7f0783c8 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -1012,7 +1012,7 @@ Glossary struct sequence A tuple with named elements. Struct sequences expose an interface similar - to :term:`named tuple` in that elements can either be accessed either by + to :term:`named tuple` in that elements can be accessed either by index or as an attribute. However, they do not have any of the named tuple methods like :meth:`~collections.somenamedtuple._make` or :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences From webhook-mailer at python.org Sat Sep 1 22:13:46 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 02 Sep 2018 02:13:46 -0000 Subject: [Python-checkins] Fix struct sequence glossary entry grammar (GH-9030) Message-ID: https://github.com/python/cpython/commit/f4c4d994ee1d49d9455f72445e9cb689a215afbf commit: f4c4d994ee1d49d9455f72445e9cb689a215afbf branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-01T22:13:43-04:00 summary: Fix struct sequence glossary entry grammar (GH-9030) ... by removing a superfluous "either". Reported by ?????? ???????? on docs at . (cherry picked from commit 98b976a2f82ba5f50cf6846338f644ca6c64f47d) Co-authored-by: Zachary Ware files: M Doc/glossary.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 12f88a9d736d..cf2ca671f26d 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -1006,7 +1006,7 @@ Glossary struct sequence A tuple with named elements. Struct sequences expose an interface similar - to :term:`named tuple` in that elements can either be accessed either by + to :term:`named tuple` in that elements can be accessed either by index or as an attribute. However, they do not have any of the named tuple methods like :meth:`~collections.somenamedtuple._make` or :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences From webhook-mailer at python.org Sat Sep 1 22:18:26 2018 From: webhook-mailer at python.org (Zachary Ware) Date: Sun, 02 Sep 2018 02:18:26 -0000 Subject: [Python-checkins] [2.7] Fix struct sequence glossary entry grammar (GH-9030) Message-ID: https://github.com/python/cpython/commit/71f2dadf66c8f9f513bb67f3b06d320c406ac2ff commit: 71f2dadf66c8f9f513bb67f3b06d320c406ac2ff branch: 2.7 author: Zachary Ware committer: GitHub date: 2018-09-01T21:18:22-05:00 summary: [2.7] Fix struct sequence glossary entry grammar (GH-9030) ... by removing a superfluous "either". Reported by ?????? ???????? on docs@ (cherry picked from commit 98b976a2f82ba5f50cf6846338f644ca6c64f47d) files: M Doc/glossary.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 34c71c6f3eee..9e6bf233bbcf 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -710,7 +710,7 @@ Glossary struct sequence A tuple with named elements. Struct sequences expose an interface similiar - to :term:`named tuple` in that elements can either be accessed either by + to :term:`named tuple` in that elements can be accessed either by index or as an attribute. However, they do not have any of the named tuple methods like :meth:`~collections.somenamedtuple._make` or :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences From solipsis at pitrou.net Sun Sep 2 05:09:23 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 02 Sep 2018 09:09:23 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=9 Message-ID: <20180902090923.1.43F73A409E5C8E39@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 3] memory blocks, sum=3 test_collections leaked [0, 7, -7] memory blocks, sum=0 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_fork leaked [-1, 2, 0] memory blocks, sum=1 test_multiprocessing_spawn leaked [-1, 0, 2] memory blocks, sum=1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflog0xUvUl', '--timeout', '7200'] From webhook-mailer at python.org Sun Sep 2 16:34:26 2018 From: webhook-mailer at python.org (Raymond Hettinger) Date: Sun, 02 Sep 2018 20:34:26 -0000 Subject: [Python-checkins] Minor improvement to code clarity (GH-9036) Message-ID: https://github.com/python/cpython/commit/f3267144269b873bcb87a9fcafe94b37be1bcfdc commit: f3267144269b873bcb87a9fcafe94b37be1bcfdc branch: master author: Raymond Hettinger committer: GitHub date: 2018-09-02T13:34:21-07:00 summary: Minor improvement to code clarity (GH-9036) Make it clear that the n==0 case is included. Otherwise, you have to know that max==0.0 whenever n==0. files: M Modules/mathmodule.c diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 8015a95d2aa7..06a969cebacb 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2074,7 +2074,7 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) if (found_nan) { return Py_NAN; } - if (max == 0.0 || n == 1) { + if (max == 0.0 || n <= 1) { return max; } for (i=0 ; i < n ; i++) { From webhook-mailer at python.org Sun Sep 2 21:48:17 2018 From: webhook-mailer at python.org (Terry Jan Reedy) Date: Mon, 03 Sep 2018 01:48:17 -0000 Subject: [Python-checkins] bpo-34500: Fix ResourceWarning in difflib.py (GH-8926) Message-ID: https://github.com/python/cpython/commit/30af2e737aad427d4da97f8dadeeecff6c2b28f5 commit: 30af2e737aad427d4da97f8dadeeecff6c2b28f5 branch: 2.7 author: Micka?l Schoentgen committer: Terry Jan Reedy date: 2018-09-02T21:48:08-04:00 summary: bpo-34500: Fix ResourceWarning in difflib.py (GH-8926) The change to Tools/scripts/diff.py effectively backports part of a2637729f23dc993e820fd92f0d1759ad714c9b2. The test code changed in Doc/library/difflib.rst is not present in current 3.x. files: A Misc/NEWS.d/next/Tools-Demos/2018-08-25-17-13-24.bpo-34500.Edz41x.rst M Doc/library/difflib.rst M Tools/scripts/diff.py diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index c6bf3ef6775d..01a3bfc2cd13 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -757,8 +757,10 @@ It is also contained in the Python source distribution, as # we're passing these as arguments to the diff function fromdate = time.ctime(os.stat(fromfile).st_mtime) todate = time.ctime(os.stat(tofile).st_mtime) - fromlines = open(fromfile, 'U').readlines() - tolines = open(tofile, 'U').readlines() + with open(fromfile, 'U') as f: + fromlines = f.readlines() + with open(tofile, 'U') as f: + tolines = f.readlines() if options.u: diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, diff --git a/Misc/NEWS.d/next/Tools-Demos/2018-08-25-17-13-24.bpo-34500.Edz41x.rst b/Misc/NEWS.d/next/Tools-Demos/2018-08-25-17-13-24.bpo-34500.Edz41x.rst new file mode 100644 index 000000000000..27ca06f1cbcd --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2018-08-25-17-13-24.bpo-34500.Edz41x.rst @@ -0,0 +1 @@ +Fix 2 ResourceWarning in difflib.py. Patch by Micka?l Schoentgen. diff --git a/Tools/scripts/diff.py b/Tools/scripts/diff.py index 513e2a7112de..c4c2e101c60e 100755 --- a/Tools/scripts/diff.py +++ b/Tools/scripts/diff.py @@ -32,8 +32,10 @@ def main(): fromdate = time.ctime(os.stat(fromfile).st_mtime) todate = time.ctime(os.stat(tofile).st_mtime) - fromlines = open(fromfile, 'U').readlines() - tolines = open(tofile, 'U').readlines() + with open(fromfile, 'U') as f: + fromlines = f.readlines() + with open(tofile, 'U') as f: + tolines = f.readlines() if options.u: diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, fromdate, todate, n=n) From solipsis at pitrou.net Mon Sep 3 05:09:17 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 03 Sep 2018 09:09:17 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=4 Message-ID: <20180903090917.1.FA8EC9172D155D61@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [7, 0, -7] memory blocks, sum=0 test_functools leaked [0, 3, 1] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogG5g4N6', '--timeout', '7200'] From webhook-mailer at python.org Mon Sep 3 08:38:24 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 12:38:24 -0000 Subject: [Python-checkins] bpo-34544: Fix setlocale() in pymain_read_conf() (GH-9041) Message-ID: https://github.com/python/cpython/commit/f01b2a1b84ee08df73a78cf1017eecf15e3cb995 commit: f01b2a1b84ee08df73a78cf1017eecf15e3cb995 branch: master author: Victor Stinner committer: GitHub date: 2018-09-03T14:38:21+02:00 summary: bpo-34544: Fix setlocale() in pymain_read_conf() (GH-9041) bpo-34485, bpo-34544: On some FreeBSD, nl_langinfo(CODESET) fails if LC_ALL or LC_CTYPE is set to an invalid locale name. Replace _Py_SetLocaleFromEnv(LC_CTYPE) with _Py_SetLocaleFromEnv(LC_ALL) to initialize properly locales. Partially revert commit 177d921c8c03d30daa32994362023f777624b10d. files: M Modules/main.c diff --git a/Modules/main.c b/Modules/main.c index 1ab555b42de4..974a0a6b78b9 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -1291,10 +1291,17 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config, int init_legacy_encoding = Py_LegacyWindowsFSEncodingFlag; #endif _PyCoreConfig save_config = _PyCoreConfig_INIT; + char *oldloc = NULL; int res = -1; - /* Set LC_CTYPE to the user preferred locale */ - _Py_SetLocaleFromEnv(LC_CTYPE); + oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL)); + if (oldloc == NULL) { + pymain->err = _Py_INIT_NO_MEMORY(); + goto done; + } + + /* Reconfigure the locale to the default for this process */ + _Py_SetLocaleFromEnv(LC_ALL); int locale_coerced = 0; int loops = 0; @@ -1385,6 +1392,10 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config, done: _PyCoreConfig_Clear(&save_config); + if (oldloc != NULL) { + setlocale(LC_ALL, oldloc); + PyMem_RawFree(oldloc); + } Py_UTF8Mode = init_utf8_mode ; #ifdef MS_WINDOWS Py_LegacyWindowsFSEncodingFlag = init_legacy_encoding; From webhook-mailer at python.org Mon Sep 3 11:05:23 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 15:05:23 -0000 Subject: [Python-checkins] _Py_CoerceLegacyLocale() restores LC_CTYPE on fail (GH-9044) Message-ID: https://github.com/python/cpython/commit/8ea09110d413829f71d979d8c7073008cb87fb03 commit: 8ea09110d413829f71d979d8c7073008cb87fb03 branch: master author: Victor Stinner committer: GitHub date: 2018-09-03T17:05:18+02:00 summary: _Py_CoerceLegacyLocale() restores LC_CTYPE on fail (GH-9044) bpo-34544: If _Py_CoerceLegacyLocale() fails to coerce the C locale, restore the LC_CTYPE locale to the its previous value. files: M Python/pylifecycle.c diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 7d17f2e65e42..33ca802fd56d 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -363,6 +363,13 @@ void _Py_CoerceLegacyLocale(int warn) { #ifdef PY_COERCE_C_LOCALE + char *oldloc = NULL; + + oldloc = _PyMem_RawStrdup(setlocale(LC_CTYPE, NULL)); + if (oldloc == NULL) { + return; + } + const char *locale_override = getenv("LC_ALL"); if (locale_override == NULL || *locale_override == '\0') { /* LC_ALL is also not set (or is set to an empty string) */ @@ -384,11 +391,16 @@ defined(HAVE_LANGINFO_H) && defined(CODESET) #endif /* Successfully configured locale, so make it the default */ _coerce_default_locale_settings(warn, target); - return; + goto done; } } } /* No C locale warning here, as Py_Initialize will emit one later */ + + setlocale(LC_CTYPE, oldloc); + +done: + PyMem_RawFree(oldloc); #endif } From webhook-mailer at python.org Mon Sep 3 11:06:42 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 15:06:42 -0000 Subject: [Python-checkins] bpo-34567: pythoninfo gets coreconfig (GH-9043) Message-ID: https://github.com/python/cpython/commit/2094c2bea4f79c31819994d8f0afa97ccc52cca8 commit: 2094c2bea4f79c31819994d8f0afa97ccc52cca8 branch: master author: Victor Stinner committer: GitHub date: 2018-09-03T17:06:39+02:00 summary: bpo-34567: pythoninfo gets coreconfig (GH-9043) * Add _testcapi.get_coreconfig() to get the _PyCoreConfig of the interpreter * test.pythoninfo now gets the core configuration using _testcapi.get_coreconfig() files: M Include/coreconfig.h M Lib/test/pythoninfo.py M Modules/_testcapimodule.c M Modules/main.c diff --git a/Include/coreconfig.h b/Include/coreconfig.h index 431c292b8164..ef043ab02df6 100644 --- a/Include/coreconfig.h +++ b/Include/coreconfig.h @@ -366,6 +366,10 @@ PyAPI_FUNC(int) _Py_SetFileSystemEncoding( PyAPI_FUNC(void) _Py_ClearFileSystemEncoding(void); #endif +#ifndef Py_LIMITED_API +PyAPI_FUNC(PyObject*) _Py_wstrlist_as_pylist(int len, wchar_t **list); +#endif + #ifdef __cplusplus } diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index f058185c2f80..21763d58c039 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -534,6 +534,17 @@ def collect_gdbm(info_add): info_add('gdbm.GDBM_VERSION', '.'.join(map(str, _GDBM_VERSION))) +def collect_get_coreconfig(info_add): + try: + from _testcapi import get_coreconfig + except ImportError: + return + + config = get_coreconfig() + for key in sorted(config): + info_add('coreconfig[%s]' % key, repr(config[key])) + + def collect_info(info): error = False info_add = info.add @@ -562,6 +573,7 @@ def collect_info(info): collect_resource, collect_cc, collect_gdbm, + collect_get_coreconfig, # Collecting from tests should be last as they have side effects. collect_test_socket, diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 7c2c57b98001..add642f223af 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -4642,6 +4642,160 @@ decode_locale_ex(PyObject *self, PyObject *args) } +static PyObject * +get_coreconfig(PyObject *self, PyObject *Py_UNUSED(args)) +{ + PyInterpreterState *interp = _PyInterpreterState_Get(); + const _PyCoreConfig *config = &interp->core_config; + PyObject *dict, *obj; + + dict = PyDict_New(); + if (dict == NULL) { + return NULL; + } + +#define FROM_STRING(STR) \ + ((STR != NULL) ? \ + PyUnicode_FromString(STR) \ + : (Py_INCREF(Py_None), Py_None)) +#define FROM_WSTRING(STR) \ + ((STR != NULL) ? \ + PyUnicode_FromWideChar(STR, -1) \ + : (Py_INCREF(Py_None), Py_None)) +#define SET_ITEM(KEY, EXPR) \ + do { \ + obj = (EXPR); \ + if (obj == NULL) { \ + return NULL; \ + } \ + int res = PyDict_SetItemString(dict, (KEY), obj); \ + Py_DECREF(obj); \ + if (res < 0) { \ + goto fail; \ + } \ + } while (0) + + SET_ITEM("install_signal_handlers", + PyLong_FromLong(config->install_signal_handlers)); + SET_ITEM("use_environment", + PyLong_FromLong(config->use_environment)); + SET_ITEM("use_hash_seed", + PyLong_FromLong(config->use_hash_seed)); + SET_ITEM("hash_seed", + PyLong_FromUnsignedLong(config->hash_seed)); + SET_ITEM("allocator", + FROM_STRING(config->allocator)); + SET_ITEM("dev_mode", + PyLong_FromLong(config->dev_mode)); + SET_ITEM("faulthandler", + PyLong_FromLong(config->faulthandler)); + SET_ITEM("tracemalloc", + PyLong_FromLong(config->tracemalloc)); + SET_ITEM("import_time", + PyLong_FromLong(config->import_time)); + SET_ITEM("show_ref_count", + PyLong_FromLong(config->show_ref_count)); + SET_ITEM("show_alloc_count", + PyLong_FromLong(config->show_alloc_count)); + SET_ITEM("dump_refs", + PyLong_FromLong(config->dump_refs)); + SET_ITEM("malloc_stats", + PyLong_FromLong(config->malloc_stats)); + SET_ITEM("coerce_c_locale", + PyLong_FromLong(config->coerce_c_locale)); + SET_ITEM("coerce_c_locale_warn", + PyLong_FromLong(config->coerce_c_locale_warn)); + SET_ITEM("filesystem_encoding", + FROM_STRING(config->filesystem_encoding)); + SET_ITEM("filesystem_errors", + FROM_STRING(config->filesystem_errors)); + SET_ITEM("stdio_encoding", + FROM_STRING(config->stdio_encoding)); + SET_ITEM("utf8_mode", + PyLong_FromLong(config->utf8_mode)); + SET_ITEM("pycache_prefix", + FROM_WSTRING(config->pycache_prefix)); + SET_ITEM("program_name", + FROM_WSTRING(config->program_name)); + SET_ITEM("argv", + _Py_wstrlist_as_pylist(config->argc, config->argv)); + SET_ITEM("program", + FROM_WSTRING(config->program)); + SET_ITEM("warnoptions", + _Py_wstrlist_as_pylist(config->nwarnoption, config->warnoptions)); + SET_ITEM("module_search_path_env", + FROM_WSTRING(config->module_search_path_env)); + SET_ITEM("home", + FROM_WSTRING(config->home)); + SET_ITEM("module_search_paths", + _Py_wstrlist_as_pylist(config->nmodule_search_path, config->module_search_paths)); + SET_ITEM("executable", + FROM_WSTRING(config->executable)); + SET_ITEM("prefix", + FROM_WSTRING(config->prefix)); + SET_ITEM("base_prefix", + FROM_WSTRING(config->base_prefix)); + SET_ITEM("exec_prefix", + FROM_WSTRING(config->exec_prefix)); + SET_ITEM("base_exec_prefix", + FROM_WSTRING(config->base_exec_prefix)); +#ifdef MS_WINDOWS + SET_ITEM("dll_path", + FROM_WSTRING(config->dll_path)); +#endif + SET_ITEM("isolated", + PyLong_FromLong(config->isolated)); + SET_ITEM("site_import", + PyLong_FromLong(config->site_import)); + SET_ITEM("bytes_warning", + PyLong_FromLong(config->bytes_warning)); + SET_ITEM("inspect", + PyLong_FromLong(config->inspect)); + SET_ITEM("interactive", + PyLong_FromLong(config->interactive)); + SET_ITEM("optimization_level", + PyLong_FromLong(config->optimization_level)); + SET_ITEM("parser_debug", + PyLong_FromLong(config->parser_debug)); + SET_ITEM("write_bytecode", + PyLong_FromLong(config->write_bytecode)); + SET_ITEM("verbose", + PyLong_FromLong(config->verbose)); + SET_ITEM("quiet", + PyLong_FromLong(config->quiet)); + SET_ITEM("user_site_directory", + PyLong_FromLong(config->user_site_directory)); + SET_ITEM("buffered_stdio", + PyLong_FromLong(config->buffered_stdio)); + SET_ITEM("stdio_encoding", + FROM_STRING(config->stdio_encoding)); + SET_ITEM("stdio_errors", + FROM_STRING(config->stdio_errors)); +#ifdef MS_WINDOWS + SET_ITEM("legacy_windows_fs_encoding", + PyLong_FromLong(config->legacy_windows_fs_encoding)); + SET_ITEM("legacy_windows_stdio", + PyLong_FromLong(config->legacy_windows_stdio)); +#endif + SET_ITEM("_install_importlib", + PyLong_FromLong(config->_install_importlib)); + SET_ITEM("_check_hash_pycs_mode", + FROM_STRING(config->_check_hash_pycs_mode)); + SET_ITEM("_frozen", + PyLong_FromLong(config->_frozen)); + + return dict; + +fail: + Py_DECREF(dict); + return NULL; + +#undef FROM_STRING +#undef FROM_WSTRING +#undef SET_ITEM +} + + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, {"raise_memoryerror", raise_memoryerror, METH_NOARGS}, @@ -4865,6 +5019,7 @@ static PyMethodDef TestMethods[] = { {"hamt", new_hamt, METH_NOARGS}, {"EncodeLocaleEx", encode_locale_ex, METH_VARARGS}, {"DecodeLocaleEx", decode_locale_ex, METH_VARARGS}, + {"get_coreconfig", get_coreconfig, METH_NOARGS}, {NULL, NULL} /* sentinel */ }; diff --git a/Modules/main.c b/Modules/main.c index 974a0a6b78b9..6f817b6c84fd 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -1018,8 +1018,8 @@ pymain_init_core_argv(_PyMain *pymain, _PyCoreConfig *config, _PyCmdline *cmdlin } -static PyObject* -wstrlist_as_pylist(int len, wchar_t **list) +PyObject* +_Py_wstrlist_as_pylist(int len, wchar_t **list) { assert(list != NULL || len < 1); @@ -1502,7 +1502,7 @@ _PyMainInterpreterConfig_Read(_PyMainInterpreterConfig *main_config, #define COPY_WSTRLIST(ATTR, LEN, LIST) \ do { \ if (ATTR == NULL) { \ - ATTR = wstrlist_as_pylist(LEN, LIST); \ + ATTR = _Py_wstrlist_as_pylist(LEN, LIST); \ if (ATTR == NULL) { \ return _Py_INIT_NO_MEMORY(); \ } \ From webhook-mailer at python.org Mon Sep 3 11:32:35 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 15:32:35 -0000 Subject: [Python-checkins] bpo-34544: pymain_read_conf() don't change LC_ALL (GH-9045) Message-ID: https://github.com/python/cpython/commit/73b00becbdd40f6a80cfa00abf1ae650a2b5e454 commit: 73b00becbdd40f6a80cfa00abf1ae650a2b5e454 branch: master author: Victor Stinner committer: GitHub date: 2018-09-03T17:32:31+02:00 summary: bpo-34544: pymain_read_conf() don't change LC_ALL (GH-9045) bpo-34485, bpo-34544: Again, pymain_read_conf() leaves LC_ALL locale unchanged: only modify LC_CTYPE. files: M Modules/main.c diff --git a/Modules/main.c b/Modules/main.c index 6f817b6c84fd..455178af8bab 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -1291,18 +1291,7 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config, int init_legacy_encoding = Py_LegacyWindowsFSEncodingFlag; #endif _PyCoreConfig save_config = _PyCoreConfig_INIT; - char *oldloc = NULL; int res = -1; - - oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL)); - if (oldloc == NULL) { - pymain->err = _Py_INIT_NO_MEMORY(); - goto done; - } - - /* Reconfigure the locale to the default for this process */ - _Py_SetLocaleFromEnv(LC_ALL); - int locale_coerced = 0; int loops = 0; @@ -1311,6 +1300,9 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config, goto done; } + /* Set LC_CTYPE to the user preferred locale */ + _Py_SetLocaleFromEnv(LC_CTYPE); + while (1) { int utf8_mode = config->utf8_mode; int encoding_changed = 0; @@ -1392,10 +1384,6 @@ pymain_read_conf(_PyMain *pymain, _PyCoreConfig *config, done: _PyCoreConfig_Clear(&save_config); - if (oldloc != NULL) { - setlocale(LC_ALL, oldloc); - PyMem_RawFree(oldloc); - } Py_UTF8Mode = init_utf8_mode ; #ifdef MS_WINDOWS Py_LegacyWindowsFSEncodingFlag = init_legacy_encoding; From webhook-mailer at python.org Mon Sep 3 16:17:11 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 20:17:11 -0000 Subject: [Python-checkins] _Py_CoerceLegacyLocale() restores LC_CTYPE on fail (GH-9044) (GH-9046) Message-ID: https://github.com/python/cpython/commit/84b0129b5e0a0e22aad22ae8db2e3833a228aa57 commit: 84b0129b5e0a0e22aad22ae8db2e3833a228aa57 branch: 3.7 author: Victor Stinner committer: GitHub date: 2018-09-03T22:17:07+02:00 summary: _Py_CoerceLegacyLocale() restores LC_CTYPE on fail (GH-9044) (GH-9046) bpo-34544: If _Py_CoerceLegacyLocale() fails to coerce the C locale, restore the LC_CTYPE locale to the its previous value. (cherry picked from commit 8ea09110d413829f71d979d8c7073008cb87fb03) files: M Python/pylifecycle.c diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 539d62a2f0f4..ba4b54864fd8 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -475,6 +475,13 @@ void _Py_CoerceLegacyLocale(const _PyCoreConfig *config) { #ifdef PY_COERCE_C_LOCALE + char *oldloc = NULL; + + oldloc = _PyMem_RawStrdup(setlocale(LC_CTYPE, NULL)); + if (oldloc == NULL) { + return; + } + const char *locale_override = getenv("LC_ALL"); if (locale_override == NULL || *locale_override == '\0') { /* LC_ALL is also not set (or is set to an empty string) */ @@ -496,11 +503,16 @@ defined(HAVE_LANGINFO_H) && defined(CODESET) #endif /* Successfully configured locale, so make it the default */ _coerce_default_locale_settings(config, target); - return; + goto done; } } } /* No C locale warning here, as Py_Initialize will emit one later */ + + setlocale(LC_CTYPE, oldloc); + +done: + PyMem_RawFree(oldloc); #endif } From webhook-mailer at python.org Mon Sep 3 17:17:24 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 21:17:24 -0000 Subject: [Python-checkins] bpo-26901: Fix the Argument Clinic test suite (GH-8879) Message-ID: https://github.com/python/cpython/commit/65fc98e7b1f62c2e621f04780a3a77c3498cc195 commit: 65fc98e7b1f62c2e621f04780a3a77c3498cc195 branch: master author: Victor Stinner committer: GitHub date: 2018-09-03T23:17:20+02:00 summary: bpo-26901: Fix the Argument Clinic test suite (GH-8879) * Fix Tools/clinic/clinic_test.py: add missing FakeClinic.destination_buffers attribute and pass a file argument to Clinic(). * Rename Tools/clinic/clinic_test.py to Lib/test/test_clinic.py: add temporary Tools/clinic/ to sys.path to import the clinic module. Co-Authored-By: Pablo Galindo files: A Lib/test/test_clinic.py D Tools/clinic/clinic_test.py diff --git a/Tools/clinic/clinic_test.py b/Lib/test/test_clinic.py similarity index 97% rename from Tools/clinic/clinic_test.py rename to Lib/test/test_clinic.py index a9479a6e44e5..ba4ae3401056 100644 --- a/Tools/clinic/clinic_test.py +++ b/Lib/test/test_clinic.py @@ -1,16 +1,26 @@ # Argument Clinic # Copyright 2012-2013 by Larry Hastings. # Licensed to the PSF under a contributor agreement. -# -import clinic -from clinic import DSLParser +from test import support +from unittest import TestCase import collections import inspect -from test import support +import os.path import sys import unittest -from unittest import TestCase + + +clinic_path = os.path.join(os.path.dirname(__file__), '..', '..', 'Tools', 'clinic') +clinic_path = os.path.normpath(clinic_path) +if not os.path.exists(clinic_path): + raise unittest.SkipTest(f'{clinic_path!r} path does not exist') +sys.path.append(clinic_path) +try: + import clinic + from clinic import DSLParser +finally: + del sys.path[-1] class FakeConverter: @@ -35,7 +45,7 @@ def get(self, name, default): return self.used_converters.setdefault(name, FakeConverterFactory(name)) clinic.Clinic.presets_text = '' -c = clinic.Clinic(language='C') +c = clinic.Clinic(language='C', filename = "file") class FakeClinic: def __init__(self): @@ -43,6 +53,7 @@ def __init__(self): self.legacy_converters = FakeConvertersDict() self.language = clinic.CLanguage(None) self.filename = None + self.destination_buffers = {} self.block_parser = clinic.BlockParser('', self.language) self.modules = collections.OrderedDict() self.classes = collections.OrderedDict() @@ -93,7 +104,7 @@ def test_eol(self): # so it would spit out an end line for you. # and since you really already had one, # the last line of the block got corrupted. - c = clinic.Clinic(clinic.CLanguage(None)) + c = clinic.Clinic(clinic.CLanguage(None), filename="file") raw = "/*[clinic]\nfoo\n[clinic]*/" cooked = c.parse(raw).splitlines() end_line = cooked[2].rstrip() @@ -252,7 +263,7 @@ def test_round_trip_2(self): def _test_clinic(self, input, output): language = clinic.CLanguage(None) - c = clinic.Clinic(language) + c = clinic.Clinic(language, filename="file") c.parsers['inert'] = InertParser(c) c.parsers['copy'] = CopyParser(c) computed = c.parse(input) From webhook-mailer at python.org Mon Sep 3 17:20:09 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Mon, 03 Sep 2018 21:20:09 -0000 Subject: [Python-checkins] bpo-33083 - Make math.factorial reject arguments that are not int-like (GH-6149) Message-ID: https://github.com/python/cpython/commit/e9ba3705de656215d52b8f8f4a2e7ad60190e944 commit: e9ba3705de656215d52b8f8f4a2e7ad60190e944 branch: master author: Pablo Galindo committer: GitHub date: 2018-09-03T22:20:06+01:00 summary: bpo-33083 - Make math.factorial reject arguments that are not int-like (GH-6149) math.factorial() was accepting non-integral Decimal instances. This is inconsistent with the actual behaviour for floats, which are not accepted. files: A Misc/NEWS.d/next/Core and Builtins/2018-03-19-00-59-20.bpo-33083.Htztjl.rst M Lib/test/test_math.py M Modules/mathmodule.c diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py index fff82fe7fadb..608789f7202f 100644 --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -5,6 +5,7 @@ from test import support import unittest import itertools +import decimal import math import os import platform @@ -510,6 +511,10 @@ def testFactorial(self): self.assertRaises(ValueError, math.factorial, -1e100) self.assertRaises(ValueError, math.factorial, math.pi) + def testFactorialNonIntegers(self): + self.assertRaises(TypeError, math.factorial, decimal.Decimal(5.2)) + self.assertRaises(TypeError, math.factorial, "5") + # Other implementations may place different upper bounds. @support.cpython_only def testFactorialHugeInputs(self): diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-03-19-00-59-20.bpo-33083.Htztjl.rst b/Misc/NEWS.d/next/Core and Builtins/2018-03-19-00-59-20.bpo-33083.Htztjl.rst new file mode 100644 index 000000000000..81df83989e48 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-03-19-00-59-20.bpo-33083.Htztjl.rst @@ -0,0 +1,2 @@ +``math.factorial`` no longer accepts arguments that are not int-like. +Patch by Pablo Galindo. diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 06a969cebacb..e872e473f5fe 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -1656,7 +1656,7 @@ math_factorial(PyObject *module, PyObject *arg) { long x; int overflow; - PyObject *result, *odd_part, *two_valuation; + PyObject *result, *odd_part, *two_valuation, *pyint_form; if (PyFloat_Check(arg)) { PyObject *lx; @@ -1672,8 +1672,14 @@ math_factorial(PyObject *module, PyObject *arg) x = PyLong_AsLongAndOverflow(lx, &overflow); Py_DECREF(lx); } - else - x = PyLong_AsLongAndOverflow(arg, &overflow); + else { + pyint_form = PyNumber_Index(arg); + if (pyint_form == NULL) { + return NULL; + } + x = PyLong_AsLongAndOverflow(pyint_form, &overflow); + Py_DECREF(pyint_form); + } if (x == -1 && PyErr_Occurred()) { return NULL; From webhook-mailer at python.org Mon Sep 3 18:24:20 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Sep 2018 22:24:20 -0000 Subject: [Python-checkins] bpo-26901: Fix the Argument Clinic test suite (GH-8879) (GH-9048) Message-ID: https://github.com/python/cpython/commit/1e921236d7ecc77299d34380d6a2159e9db05a1a commit: 1e921236d7ecc77299d34380d6a2159e9db05a1a branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-04T00:24:17+02:00 summary: bpo-26901: Fix the Argument Clinic test suite (GH-8879) (GH-9048) * Fix Tools/clinic/clinic_test.py: add missing FakeClinic.destination_buffers attribute and pass a file argument to Clinic(). * Rename Tools/clinic/clinic_test.py to Lib/test/test_clinic.py: add temporary Tools/clinic/ to sys.path to import the clinic module. Co-Authored-By: Pablo Galindo (cherry picked from commit 65fc98e7b1f62c2e621f04780a3a77c3498cc195) Co-authored-by: Victor Stinner files: A Lib/test/test_clinic.py D Tools/clinic/clinic_test.py diff --git a/Tools/clinic/clinic_test.py b/Lib/test/test_clinic.py similarity index 97% rename from Tools/clinic/clinic_test.py rename to Lib/test/test_clinic.py index a9479a6e44e5..ba4ae3401056 100644 --- a/Tools/clinic/clinic_test.py +++ b/Lib/test/test_clinic.py @@ -1,16 +1,26 @@ # Argument Clinic # Copyright 2012-2013 by Larry Hastings. # Licensed to the PSF under a contributor agreement. -# -import clinic -from clinic import DSLParser +from test import support +from unittest import TestCase import collections import inspect -from test import support +import os.path import sys import unittest -from unittest import TestCase + + +clinic_path = os.path.join(os.path.dirname(__file__), '..', '..', 'Tools', 'clinic') +clinic_path = os.path.normpath(clinic_path) +if not os.path.exists(clinic_path): + raise unittest.SkipTest(f'{clinic_path!r} path does not exist') +sys.path.append(clinic_path) +try: + import clinic + from clinic import DSLParser +finally: + del sys.path[-1] class FakeConverter: @@ -35,7 +45,7 @@ def get(self, name, default): return self.used_converters.setdefault(name, FakeConverterFactory(name)) clinic.Clinic.presets_text = '' -c = clinic.Clinic(language='C') +c = clinic.Clinic(language='C', filename = "file") class FakeClinic: def __init__(self): @@ -43,6 +53,7 @@ def __init__(self): self.legacy_converters = FakeConvertersDict() self.language = clinic.CLanguage(None) self.filename = None + self.destination_buffers = {} self.block_parser = clinic.BlockParser('', self.language) self.modules = collections.OrderedDict() self.classes = collections.OrderedDict() @@ -93,7 +104,7 @@ def test_eol(self): # so it would spit out an end line for you. # and since you really already had one, # the last line of the block got corrupted. - c = clinic.Clinic(clinic.CLanguage(None)) + c = clinic.Clinic(clinic.CLanguage(None), filename="file") raw = "/*[clinic]\nfoo\n[clinic]*/" cooked = c.parse(raw).splitlines() end_line = cooked[2].rstrip() @@ -252,7 +263,7 @@ def test_round_trip_2(self): def _test_clinic(self, input, output): language = clinic.CLanguage(None) - c = clinic.Clinic(language) + c = clinic.Clinic(language, filename="file") c.parsers['inert'] = InertParser(c) c.parsers['copy'] = CopyParser(c) computed = c.parse(input) From webhook-mailer at python.org Tue Sep 4 04:53:59 2018 From: webhook-mailer at python.org (Antoine Pitrou) Date: Tue, 04 Sep 2018 08:53:59 -0000 Subject: [Python-checkins] bpo-33613, test_semaphore_tracker_sigint: fix race condition (#7850) Message-ID: https://github.com/python/cpython/commit/ec74d187f50a8a48f94eb37023300583fbd644cc commit: ec74d187f50a8a48f94eb37023300583fbd644cc branch: master author: Pablo Galindo committer: Antoine Pitrou date: 2018-09-04T10:53:54+02:00 summary: bpo-33613, test_semaphore_tracker_sigint: fix race condition (#7850) Fail `test_semaphore_tracker_sigint` if no warnings are expected and one is received. Fix race condition when the child receives SIGINT before it can register signal handlers for it. The race condition occurs when the parent calls `_semaphore_tracker.ensure_running()` (which in turn spawns the semaphore_tracker using `_posixsubprocess.fork_exec`), the child registers the signal handlers and the parent tries to kill the child. What seem to happen is that in some slow systems, the parent sends the signal to kill the child before the child protects against the signal. files: A Misc/NEWS.d/next/Library/2018-07-31-23-33-06.bpo-33613.Cdnt0i.rst M Lib/multiprocessing/semaphore_tracker.py M Lib/test/_test_multiprocessing.py diff --git a/Lib/multiprocessing/semaphore_tracker.py b/Lib/multiprocessing/semaphore_tracker.py index 3e31bf8402ee..82833bcf861a 100644 --- a/Lib/multiprocessing/semaphore_tracker.py +++ b/Lib/multiprocessing/semaphore_tracker.py @@ -23,6 +23,9 @@ __all__ = ['ensure_running', 'register', 'unregister'] +_HAVE_SIGMASK = hasattr(signal, 'pthread_sigmask') +_IGNORED_SIGNALS = (signal.SIGINT, signal.SIGTERM) + class SemaphoreTracker(object): @@ -43,10 +46,16 @@ def ensure_running(self): with self._lock: if self._pid is not None: # semaphore tracker was launched before, is it still running? - pid, status = os.waitpid(self._pid, os.WNOHANG) - if not pid: - # => still alive - return + try: + pid, _ = os.waitpid(self._pid, os.WNOHANG) + except ChildProcessError: + # The process terminated + pass + else: + if not pid: + # => still alive + return + # => dead, launch it again os.close(self._fd) self._fd = None @@ -68,7 +77,19 @@ def ensure_running(self): exe = spawn.get_executable() args = [exe] + util._args_from_interpreter_flags() args += ['-c', cmd % r] - pid = util.spawnv_passfds(exe, args, fds_to_pass) + # bpo-33613: Register a signal mask that will block the signals. + # This signal mask will be inherited by the child that is going + # to be spawned and will protect the child from a race condition + # that can make the child die before it registers signal handlers + # for SIGINT and SIGTERM. The mask is unregistered after spawning + # the child. + try: + if _HAVE_SIGMASK: + signal.pthread_sigmask(signal.SIG_BLOCK, _IGNORED_SIGNALS) + pid = util.spawnv_passfds(exe, args, fds_to_pass) + finally: + if _HAVE_SIGMASK: + signal.pthread_sigmask(signal.SIG_UNBLOCK, _IGNORED_SIGNALS) except: os.close(w) raise @@ -104,12 +125,13 @@ def _send(self, cmd, name): unregister = _semaphore_tracker.unregister getfd = _semaphore_tracker.getfd - def main(fd): '''Run semaphore tracker.''' # protect the process from ^C and "killall python" etc signal.signal(signal.SIGINT, signal.SIG_IGN) signal.signal(signal.SIGTERM, signal.SIG_IGN) + if _HAVE_SIGMASK: + signal.pthread_sigmask(signal.SIG_UNBLOCK, _IGNORED_SIGNALS) for f in (sys.stdin, sys.stdout): try: diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 749cf8c4fd2c..a5509ce9135a 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -20,6 +20,7 @@ import struct import operator import weakref +import warnings import test.support import test.support.script_helper from test import support @@ -4517,17 +4518,19 @@ def check_semaphore_tracker_death(self, signum, should_die): # bpo-31310: if the semaphore tracker process has died, it should # be restarted implicitly. from multiprocessing.semaphore_tracker import _semaphore_tracker - _semaphore_tracker.ensure_running() pid = _semaphore_tracker._pid + if pid is not None: + os.kill(pid, signal.SIGKILL) + os.waitpid(pid, 0) + with warnings.catch_warnings(record=True) as all_warn: + _semaphore_tracker.ensure_running() + pid = _semaphore_tracker._pid + os.kill(pid, signum) time.sleep(1.0) # give it time to die ctx = multiprocessing.get_context("spawn") - with contextlib.ExitStack() as stack: - if should_die: - stack.enter_context(self.assertWarnsRegex( - UserWarning, - "semaphore_tracker: process died")) + with warnings.catch_warnings(record=True) as all_warn: sem = ctx.Semaphore() sem.acquire() sem.release() @@ -4537,11 +4540,23 @@ def check_semaphore_tracker_death(self, signum, should_die): del sem gc.collect() self.assertIsNone(wr()) + if should_die: + self.assertEqual(len(all_warn), 1) + the_warn = all_warn[0] + issubclass(the_warn.category, UserWarning) + self.assertTrue("semaphore_tracker: process died" + in str(the_warn.message)) + else: + self.assertEqual(len(all_warn), 0) def test_semaphore_tracker_sigint(self): # Catchable signal (ignored by semaphore tracker) self.check_semaphore_tracker_death(signal.SIGINT, False) + def test_semaphore_tracker_sigterm(self): + # Catchable signal (ignored by semaphore tracker) + self.check_semaphore_tracker_death(signal.SIGTERM, False) + def test_semaphore_tracker_sigkill(self): # Uncatchable signal. self.check_semaphore_tracker_death(signal.SIGKILL, True) diff --git a/Misc/NEWS.d/next/Library/2018-07-31-23-33-06.bpo-33613.Cdnt0i.rst b/Misc/NEWS.d/next/Library/2018-07-31-23-33-06.bpo-33613.Cdnt0i.rst new file mode 100644 index 000000000000..9574e43fcc60 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-07-31-23-33-06.bpo-33613.Cdnt0i.rst @@ -0,0 +1,3 @@ +Fix a race condition in ``multiprocessing.semaphore_tracker`` when the +tracker receives SIGINT before it can register signal handlers for ignoring +it. From webhook-mailer at python.org Tue Sep 4 05:01:13 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 04 Sep 2018 09:01:13 -0000 Subject: [Python-checkins] bpo-34530: Fix distutils find_executable() (GH-9049) Message-ID: https://github.com/python/cpython/commit/39487196c87e28128ea907a0d9b8a88ba53f68d5 commit: 39487196c87e28128ea907a0d9b8a88ba53f68d5 branch: master author: Victor Stinner committer: GitHub date: 2018-09-04T11:01:09+02:00 summary: bpo-34530: Fix distutils find_executable() (GH-9049) distutils.spawn.find_executable() now falls back on os.defpath if the PATH environment variable is not set. files: A Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst M Lib/distutils/spawn.py M Lib/distutils/tests/test_spawn.py diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py index 5dd415a283d1..538768809327 100644 --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -173,7 +173,7 @@ def find_executable(executable, path=None): os.environ['PATH']. Returns the complete filename or None if not found. """ if path is None: - path = os.environ['PATH'] + path = os.environ.get('PATH', os.defpath) paths = path.split(os.pathsep) base, ext = os.path.splitext(executable) diff --git a/Lib/distutils/tests/test_spawn.py b/Lib/distutils/tests/test_spawn.py index 5edc24a3a104..0d455385d8ac 100644 --- a/Lib/distutils/tests/test_spawn.py +++ b/Lib/distutils/tests/test_spawn.py @@ -1,9 +1,13 @@ """Tests for distutils.spawn.""" -import unittest -import sys import os +import stat +import sys +import unittest +from unittest import mock from test.support import run_unittest, unix_shell +from test import support as test_support +from distutils.spawn import find_executable from distutils.spawn import _nt_quote_args from distutils.spawn import spawn from distutils.errors import DistutilsExecError @@ -51,6 +55,47 @@ def test_spawn(self): os.chmod(exe, 0o777) spawn([exe]) # should work without any error + def test_find_executable(self): + with test_support.temp_dir() as tmp_dir: + # use TESTFN to get a pseudo-unique filename + program_noeext = test_support.TESTFN + # Give the temporary program an ".exe" suffix for all. + # It's needed on Windows and not harmful on other platforms. + program = program_noeext + ".exe" + + filename = os.path.join(tmp_dir, program) + with open(filename, "wb"): + pass + os.chmod(filename, stat.S_IXUSR) + + # test path parameter + rv = find_executable(program, path=tmp_dir) + self.assertEqual(rv, filename) + + if sys.platform == 'win32': + # test without ".exe" extension + rv = find_executable(program_noeext, path=tmp_dir) + self.assertEqual(rv, filename) + + # test find in the current directory + with test_support.change_cwd(tmp_dir): + rv = find_executable(program) + self.assertEqual(rv, program) + + # test non-existent program + dont_exist_program = "dontexist_" + program + rv = find_executable(dont_exist_program , path=tmp_dir) + self.assertIsNone(rv) + + # test os.defpath: missing PATH environment variable + with test_support.EnvironmentVarGuard() as env: + with mock.patch('distutils.spawn.os.defpath', tmp_dir): + env.pop('PATH') + + rv = find_executable(program) + self.assertEqual(rv, filename) + + def test_suite(): return unittest.makeSuite(SpawnTestCase) diff --git a/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst new file mode 100644 index 000000000000..064de73c0ffe --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst @@ -0,0 +1,2 @@ +``distutils.spawn.find_executable()`` now falls back on :data:`os.defpath` +if the ``PATH`` environment variable is not set. From solipsis at pitrou.net Tue Sep 4 05:08:26 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 04 Sep 2018 09:08:26 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-2 Message-ID: <20180904090826.1.A6991F1C79116EEC@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [-7, 1, 0] memory blocks, sum=-6 test_functools leaked [0, 3, 1] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogxyXL6_', '--timeout', '7200'] From webhook-mailer at python.org Tue Sep 4 05:19:20 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 04 Sep 2018 09:19:20 -0000 Subject: [Python-checkins] bpo-34530: Fix distutils find_executable() (GH-9049) Message-ID: https://github.com/python/cpython/commit/7aa3eadca2a50ce651ce603d6100b05bb7106f1c commit: 7aa3eadca2a50ce651ce603d6100b05bb7106f1c branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-04T05:19:13-04:00 summary: bpo-34530: Fix distutils find_executable() (GH-9049) distutils.spawn.find_executable() now falls back on os.defpath if the PATH environment variable is not set. (cherry picked from commit 39487196c87e28128ea907a0d9b8a88ba53f68d5) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst M Lib/distutils/spawn.py M Lib/distutils/tests/test_spawn.py diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py index 5dd415a283d1..538768809327 100644 --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -173,7 +173,7 @@ def find_executable(executable, path=None): os.environ['PATH']. Returns the complete filename or None if not found. """ if path is None: - path = os.environ['PATH'] + path = os.environ.get('PATH', os.defpath) paths = path.split(os.pathsep) base, ext = os.path.splitext(executable) diff --git a/Lib/distutils/tests/test_spawn.py b/Lib/distutils/tests/test_spawn.py index 5edc24a3a104..0d455385d8ac 100644 --- a/Lib/distutils/tests/test_spawn.py +++ b/Lib/distutils/tests/test_spawn.py @@ -1,9 +1,13 @@ """Tests for distutils.spawn.""" -import unittest -import sys import os +import stat +import sys +import unittest +from unittest import mock from test.support import run_unittest, unix_shell +from test import support as test_support +from distutils.spawn import find_executable from distutils.spawn import _nt_quote_args from distutils.spawn import spawn from distutils.errors import DistutilsExecError @@ -51,6 +55,47 @@ def test_spawn(self): os.chmod(exe, 0o777) spawn([exe]) # should work without any error + def test_find_executable(self): + with test_support.temp_dir() as tmp_dir: + # use TESTFN to get a pseudo-unique filename + program_noeext = test_support.TESTFN + # Give the temporary program an ".exe" suffix for all. + # It's needed on Windows and not harmful on other platforms. + program = program_noeext + ".exe" + + filename = os.path.join(tmp_dir, program) + with open(filename, "wb"): + pass + os.chmod(filename, stat.S_IXUSR) + + # test path parameter + rv = find_executable(program, path=tmp_dir) + self.assertEqual(rv, filename) + + if sys.platform == 'win32': + # test without ".exe" extension + rv = find_executable(program_noeext, path=tmp_dir) + self.assertEqual(rv, filename) + + # test find in the current directory + with test_support.change_cwd(tmp_dir): + rv = find_executable(program) + self.assertEqual(rv, program) + + # test non-existent program + dont_exist_program = "dontexist_" + program + rv = find_executable(dont_exist_program , path=tmp_dir) + self.assertIsNone(rv) + + # test os.defpath: missing PATH environment variable + with test_support.EnvironmentVarGuard() as env: + with mock.patch('distutils.spawn.os.defpath', tmp_dir): + env.pop('PATH') + + rv = find_executable(program) + self.assertEqual(rv, filename) + + def test_suite(): return unittest.makeSuite(SpawnTestCase) diff --git a/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst new file mode 100644 index 000000000000..064de73c0ffe --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst @@ -0,0 +1,2 @@ +``distutils.spawn.find_executable()`` now falls back on :data:`os.defpath` +if the ``PATH`` environment variable is not set. From webhook-mailer at python.org Tue Sep 4 05:40:35 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 04 Sep 2018 09:40:35 -0000 Subject: [Python-checkins] bpo-34530: Fix distutils find_executable() (GH-9049) (GH-9058) Message-ID: https://github.com/python/cpython/commit/7056ca880bf4ff428dfb2c4eee67919dc43ecd34 commit: 7056ca880bf4ff428dfb2c4eee67919dc43ecd34 branch: 2.7 author: Victor Stinner committer: GitHub date: 2018-09-04T11:40:29+02:00 summary: bpo-34530: Fix distutils find_executable() (GH-9049) (GH-9058) distutils.spawn.find_executable() now falls back on os.defpath if the PATH environment variable is not set. (cherry picked from commit 39487196c87e28128ea907a0d9b8a88ba53f68d5) files: A Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst M Lib/distutils/spawn.py M Lib/distutils/tests/test_spawn.py diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py index 321344a3a583..737b293f8db3 100644 --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -208,7 +208,8 @@ def find_executable(executable, path=None): os.environ['PATH']. Returns the complete filename or None if not found. """ if path is None: - path = os.environ['PATH'] + path = os.environ.get('PATH', os.defpath) + paths = path.split(os.pathsep) base, ext = os.path.splitext(executable) diff --git a/Lib/distutils/tests/test_spawn.py b/Lib/distutils/tests/test_spawn.py index defa54d87f12..061a72f1a51d 100644 --- a/Lib/distutils/tests/test_spawn.py +++ b/Lib/distutils/tests/test_spawn.py @@ -1,8 +1,11 @@ """Tests for distutils.spawn.""" -import unittest import os +import stat +import sys import time -from test.test_support import captured_stdout, run_unittest +import unittest +from test.support import captured_stdout, run_unittest +from test import support as test_support from distutils.spawn import _nt_quote_args from distutils.spawn import spawn, find_executable @@ -53,6 +56,48 @@ def test_spawn(self): os.chmod(exe, 0777) spawn([exe]) # should work without any error + def test_find_executable(self): + with test_support.temp_dir() as tmp_dir: + # use TESTFN to get a pseudo-unique filename + program_noeext = test_support.TESTFN + # Give the temporary program an ".exe" suffix for all. + # It's needed on Windows and not harmful on other platforms. + program = program_noeext + ".exe" + + filename = os.path.join(tmp_dir, program) + with open(filename, "wb"): + pass + os.chmod(filename, stat.S_IXUSR) + + # test path parameter + rv = find_executable(program, path=tmp_dir) + self.assertEqual(rv, filename) + + if sys.platform == 'win32': + # test without ".exe" extension + rv = find_executable(program_noeext, path=tmp_dir) + self.assertEqual(rv, filename) + + # test find in the current directory + with test_support.change_cwd(tmp_dir): + rv = find_executable(program) + self.assertEqual(rv, program) + + # test non-existent program + dont_exist_program = "dontexist_" + program + rv = find_executable(dont_exist_program , path=tmp_dir) + self.assertIsNone(rv) + + # test os.defpath: missing PATH environment variable + with test_support.EnvironmentVarGuard() as env: + from distutils import spawn + with test_support.swap_attr(spawn.os, 'defpath', tmp_dir): + env.pop('PATH') + + rv = find_executable(program) + self.assertEqual(rv, filename) + + def test_suite(): return unittest.makeSuite(SpawnTestCase) diff --git a/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst new file mode 100644 index 000000000000..064de73c0ffe --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst @@ -0,0 +1,2 @@ +``distutils.spawn.find_executable()`` now falls back on :data:`os.defpath` +if the ``PATH`` environment variable is not set. From webhook-mailer at python.org Tue Sep 4 08:04:36 2018 From: webhook-mailer at python.org (Serhiy Storchaka) Date: Tue, 04 Sep 2018 12:04:36 -0000 Subject: [Python-checkins] bpo-26544: Add test for platform._comparable_version(). (GH-8973) Message-ID: https://github.com/python/cpython/commit/7917aadb3edb7616d6164c5eaba24df6ac0a5fc6 commit: 7917aadb3edb7616d6164c5eaba24df6ac0a5fc6 branch: master author: Serhiy Storchaka committer: GitHub date: 2018-09-04T15:04:25+03:00 summary: bpo-26544: Add test for platform._comparable_version(). (GH-8973) files: M Lib/test/test_platform.py diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py index 9ecd5d904e30..fd6da313206d 100644 --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -275,6 +275,42 @@ def test_libc_ver(self): self.assertEqual(platform.libc_ver(support.TESTFN), ('glibc', '1.23.4')) + @support.cpython_only + def test__comparable_version(self): + from platform import _comparable_version as V + self.assertEqual(V('1.2.3'), V('1.2.3')) + self.assertLess(V('1.2.3'), V('1.2.10')) + self.assertEqual(V('1.2.3.4'), V('1_2-3+4')) + self.assertLess(V('1.2spam'), V('1.2dev')) + self.assertLess(V('1.2dev'), V('1.2alpha')) + self.assertLess(V('1.2dev'), V('1.2a')) + self.assertLess(V('1.2alpha'), V('1.2beta')) + self.assertLess(V('1.2a'), V('1.2b')) + self.assertLess(V('1.2beta'), V('1.2c')) + self.assertLess(V('1.2b'), V('1.2c')) + self.assertLess(V('1.2c'), V('1.2RC')) + self.assertLess(V('1.2c'), V('1.2rc')) + self.assertLess(V('1.2RC'), V('1.2.0')) + self.assertLess(V('1.2rc'), V('1.2.0')) + self.assertLess(V('1.2.0'), V('1.2pl')) + self.assertLess(V('1.2.0'), V('1.2p')) + + self.assertLess(V('1.5.1'), V('1.5.2b2')) + self.assertLess(V('3.10a'), V('161')) + self.assertEqual(V('8.02'), V('8.02')) + self.assertLess(V('3.4j'), V('1996.07.12')) + self.assertLess(V('3.1.1.6'), V('3.2.pl0')) + self.assertLess(V('2g6'), V('11g')) + self.assertLess(V('0.9'), V('2.2')) + self.assertLess(V('1.2'), V('1.2.1')) + self.assertLess(V('1.1'), V('1.2.2')) + self.assertLess(V('1.1'), V('1.2')) + self.assertLess(V('1.2.1'), V('1.2.2')) + self.assertLess(V('1.2'), V('1.2.2')) + self.assertLess(V('0.4'), V('0.4.0')) + self.assertLess(V('1.13++'), V('5.5.kw')) + self.assertLess(V('0.960923'), V('2.2beta29')) + def test_popen(self): mswindows = (sys.platform == "win32") From webhook-mailer at python.org Tue Sep 4 10:31:26 2018 From: webhook-mailer at python.org (Serhiy Storchaka) Date: Tue, 04 Sep 2018 14:31:26 -0000 Subject: [Python-checkins] [3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). (GH-8970) Message-ID: https://github.com/python/cpython/commit/20a8392cec2967f15ae81633c1775645b3ca40da commit: 20a8392cec2967f15ae81633c1775645b3ca40da branch: 3.7 author: Serhiy Storchaka committer: GitHub date: 2018-09-04T17:31:18+03:00 summary: [3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356). (GH-8970) (cherry picked from commit 7d81e8f5995df6980a1a02923e224a481375f130) files: M Lib/platform.py M Lib/test/test_platform.py diff --git a/Lib/platform.py b/Lib/platform.py index e22734850786..4482f479bfde 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -136,6 +136,35 @@ # Constant used by test_platform to test linux_distribution(). _UNIXCONFDIR = '/etc' +# Helper for comparing two version number strings. +# Based on the description of the PHP's version_compare(): +# http://php.net/manual/en/function.version-compare.php + +_ver_stages = { + # any string not found in this dict, will get 0 assigned + 'dev': 10, + 'alpha': 20, 'a': 20, + 'beta': 30, 'b': 30, + 'c': 40, + 'RC': 50, 'rc': 50, + # number, will get 100 assigned + 'pl': 200, 'p': 200, +} + +_component_re = re.compile(r'([0-9]+|[._+-])') + +def _comparable_version(version): + result = [] + for v in _component_re.split(version): + if v not in '._+-': + try: + v = int(v, 10) + t = 100 + except ValueError: + t = _ver_stages.get(v, 0) + result.extend((t, v)) + return result + ### Platform specific APIs _libc_search = re.compile(b'(__libc_init)' @@ -159,7 +188,7 @@ def libc_ver(executable=sys.executable, lib='', version='', chunksize=16384): The file is read and scanned in chunks of chunksize bytes. """ - from distutils.version import LooseVersion as V + V = _comparable_version if hasattr(os.path, 'realpath'): # Python 2.2 introduced os.path.realpath(); it is used # here to work around problems with Cygwin not being diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py index eaf14e0f18ad..509d8865c068 100644 --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -285,6 +285,42 @@ def test_libc_ver(self): self.assertEqual(platform.libc_ver(support.TESTFN), ('glibc', '1.23.4')) + @support.cpython_only + def test__comparable_version(self): + from platform import _comparable_version as V + self.assertEqual(V('1.2.3'), V('1.2.3')) + self.assertLess(V('1.2.3'), V('1.2.10')) + self.assertEqual(V('1.2.3.4'), V('1_2-3+4')) + self.assertLess(V('1.2spam'), V('1.2dev')) + self.assertLess(V('1.2dev'), V('1.2alpha')) + self.assertLess(V('1.2dev'), V('1.2a')) + self.assertLess(V('1.2alpha'), V('1.2beta')) + self.assertLess(V('1.2a'), V('1.2b')) + self.assertLess(V('1.2beta'), V('1.2c')) + self.assertLess(V('1.2b'), V('1.2c')) + self.assertLess(V('1.2c'), V('1.2RC')) + self.assertLess(V('1.2c'), V('1.2rc')) + self.assertLess(V('1.2RC'), V('1.2.0')) + self.assertLess(V('1.2rc'), V('1.2.0')) + self.assertLess(V('1.2.0'), V('1.2pl')) + self.assertLess(V('1.2.0'), V('1.2p')) + + self.assertLess(V('1.5.1'), V('1.5.2b2')) + self.assertLess(V('3.10a'), V('161')) + self.assertEqual(V('8.02'), V('8.02')) + self.assertLess(V('3.4j'), V('1996.07.12')) + self.assertLess(V('3.1.1.6'), V('3.2.pl0')) + self.assertLess(V('2g6'), V('11g')) + self.assertLess(V('0.9'), V('2.2')) + self.assertLess(V('1.2'), V('1.2.1')) + self.assertLess(V('1.1'), V('1.2.2')) + self.assertLess(V('1.1'), V('1.2')) + self.assertLess(V('1.2.1'), V('1.2.2')) + self.assertLess(V('1.2'), V('1.2.2')) + self.assertLess(V('0.4'), V('0.4.0')) + self.assertLess(V('1.13++'), V('5.5.kw')) + self.assertLess(V('0.960923'), V('2.2beta29')) + def test_parse_release_file(self): for input, output in ( From webhook-mailer at python.org Tue Sep 4 12:10:37 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 04 Sep 2018 16:10:37 -0000 Subject: [Python-checkins] bpo-34563: Fix for invalid assert on big output of multiprocessing.Process (GH-9027) Message-ID: https://github.com/python/cpython/commit/266f4904a222a784080e29aad0916849e507515d commit: 266f4904a222a784080e29aad0916849e507515d branch: master author: Alexander Buchkovsky committer: Victor Stinner date: 2018-09-04T18:10:28+02:00 summary: bpo-34563: Fix for invalid assert on big output of multiprocessing.Process (GH-9027) Fix for invalid assert on big output of multiprocessing.Process. files: A Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst M Modules/_winapi.c M Modules/clinic/_winapi.c.h diff --git a/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst b/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst new file mode 100644 index 000000000000..9127af0d1924 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst @@ -0,0 +1 @@ +On Windows, fix multiprocessing.Connection for very large read: fix _winapi.PeekNamedPipe() and _winapi.ReadFile() for read larger than INT_MAX (usually 2^31-1). \ No newline at end of file diff --git a/Modules/_winapi.c b/Modules/_winapi.c index 30f269fbdccd..44788e5992a0 100644 --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -1338,7 +1338,7 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) } if (_PyBytes_Resize(&buf, nread)) return NULL; - return Py_BuildValue("Nii", buf, navail, nleft); + return Py_BuildValue("NII", buf, navail, nleft); } else { Py_BEGIN_ALLOW_THREADS @@ -1347,7 +1347,7 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) if (!ret) { return PyErr_SetExcFromWindowsErr(PyExc_OSError, 0); } - return Py_BuildValue("ii", navail, nleft); + return Py_BuildValue("II", navail, nleft); } } @@ -1355,14 +1355,14 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) _winapi.ReadFile handle: HANDLE - size: int + size: DWORD overlapped as use_overlapped: bool(accept={int}) = False [clinic start generated code]*/ static PyObject * -_winapi_ReadFile_impl(PyObject *module, HANDLE handle, int size, +_winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, int use_overlapped) -/*[clinic end generated code: output=492029ca98161d84 input=3f0fde92f74de59a]*/ +/*[clinic end generated code: output=d3d5b44a8201b944 input=08c439d03a11aac5]*/ { DWORD nread; PyObject *buf; diff --git a/Modules/clinic/_winapi.c.h b/Modules/clinic/_winapi.c.h index b14f087732ef..c66522ebab6d 100644 --- a/Modules/clinic/_winapi.c.h +++ b/Modules/clinic/_winapi.c.h @@ -673,7 +673,7 @@ PyDoc_STRVAR(_winapi_ReadFile__doc__, {"ReadFile", (PyCFunction)_winapi_ReadFile, METH_FASTCALL|METH_KEYWORDS, _winapi_ReadFile__doc__}, static PyObject * -_winapi_ReadFile_impl(PyObject *module, HANDLE handle, int size, +_winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, int use_overlapped); static PyObject * @@ -681,9 +681,9 @@ _winapi_ReadFile(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyOb { PyObject *return_value = NULL; static const char * const _keywords[] = {"handle", "size", "overlapped", NULL}; - static _PyArg_Parser _parser = {"" F_HANDLE "i|i:ReadFile", _keywords, 0}; + static _PyArg_Parser _parser = {"" F_HANDLE "k|i:ReadFile", _keywords, 0}; HANDLE handle; - int size; + DWORD size; int use_overlapped = 0; if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, @@ -941,4 +941,4 @@ _winapi_GetFileType(PyObject *module, PyObject *const *args, Py_ssize_t nargs, P exit: return return_value; } -/*[clinic end generated code: output=ec7f36eb7efc9d00 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=baaf3d379b91be0a input=a9049054013a1b77]*/ From webhook-mailer at python.org Tue Sep 4 12:35:55 2018 From: webhook-mailer at python.org (Steve Dower) Date: Tue, 04 Sep 2018 16:35:55 -0000 Subject: [Python-checkins] bpo-34565: Change a PC/launcher.c comment to accurately describe valid major versions. (GH-9037) Message-ID: https://github.com/python/cpython/commit/3876af4f7c2ef87db6d2d83efc229955968926dd commit: 3876af4f7c2ef87db6d2d83efc229955968926dd branch: master author: Brendan Gerrity committer: Steve Dower date: 2018-09-04T09:35:46-07:00 summary: bpo-34565: Change a PC/launcher.c comment to accurately describe valid major versions. (GH-9037) files: M PC/launcher.c diff --git a/PC/launcher.c b/PC/launcher.c index fcc3abb63de6..75a06c2310c8 100644 --- a/PC/launcher.c +++ b/PC/launcher.c @@ -1051,7 +1051,7 @@ static BOOL validate_version(wchar_t * p) { /* - Version information should start with one of 2 or 3, + Version information should start with the major version, Optionally followed by a period and a minor version, Optionally followed by a minus and one of 32 or 64. Valid examples: @@ -1068,7 +1068,7 @@ validate_version(wchar_t * p) */ BOOL result = (p != NULL); /* Default to False if null pointer. */ - result = result && iswdigit(*p); /* Result = False if fist string element is not a digit. */ + result = result && iswdigit(*p); /* Result = False if first string element is not a digit. */ while (result && iswdigit(*p)) /* Require a major version */ ++p; /* Skip all leading digit(s) */ From webhook-mailer at python.org Tue Sep 4 15:40:08 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 04 Sep 2018 19:40:08 -0000 Subject: [Python-checkins] bpo-34563: Fix for invalid assert on big output of multiprocessing.Process (GH-9027) (GH-9064) Message-ID: https://github.com/python/cpython/commit/e8ca8806a9f47b3bac4ae1c6b8a54c47d1aad8f3 commit: e8ca8806a9f47b3bac4ae1c6b8a54c47d1aad8f3 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-04T21:39:54+02:00 summary: bpo-34563: Fix for invalid assert on big output of multiprocessing.Process (GH-9027) (GH-9064) Fix for invalid assert on big output of multiprocessing.Process. (cherry picked from commit 266f4904a222a784080e29aad0916849e507515d) Co-authored-by: Alexander Buchkovsky files: A Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst M Modules/_winapi.c M Modules/clinic/_winapi.c.h diff --git a/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst b/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst new file mode 100644 index 000000000000..9127af0d1924 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst @@ -0,0 +1 @@ +On Windows, fix multiprocessing.Connection for very large read: fix _winapi.PeekNamedPipe() and _winapi.ReadFile() for read larger than INT_MAX (usually 2^31-1). \ No newline at end of file diff --git a/Modules/_winapi.c b/Modules/_winapi.c index c596cba3cbc3..9ea5a9266295 100644 --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -1337,7 +1337,7 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) } if (_PyBytes_Resize(&buf, nread)) return NULL; - return Py_BuildValue("Nii", buf, navail, nleft); + return Py_BuildValue("NII", buf, navail, nleft); } else { Py_BEGIN_ALLOW_THREADS @@ -1346,7 +1346,7 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) if (!ret) { return PyErr_SetExcFromWindowsErr(PyExc_OSError, 0); } - return Py_BuildValue("ii", navail, nleft); + return Py_BuildValue("II", navail, nleft); } } @@ -1354,14 +1354,14 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) _winapi.ReadFile handle: HANDLE - size: int + size: DWORD overlapped as use_overlapped: bool(accept={int}) = False [clinic start generated code]*/ static PyObject * -_winapi_ReadFile_impl(PyObject *module, HANDLE handle, int size, +_winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, int use_overlapped) -/*[clinic end generated code: output=492029ca98161d84 input=3f0fde92f74de59a]*/ +/*[clinic end generated code: output=d3d5b44a8201b944 input=08c439d03a11aac5]*/ { DWORD nread; PyObject *buf; diff --git a/Modules/clinic/_winapi.c.h b/Modules/clinic/_winapi.c.h index b14f087732ef..c66522ebab6d 100644 --- a/Modules/clinic/_winapi.c.h +++ b/Modules/clinic/_winapi.c.h @@ -673,7 +673,7 @@ PyDoc_STRVAR(_winapi_ReadFile__doc__, {"ReadFile", (PyCFunction)_winapi_ReadFile, METH_FASTCALL|METH_KEYWORDS, _winapi_ReadFile__doc__}, static PyObject * -_winapi_ReadFile_impl(PyObject *module, HANDLE handle, int size, +_winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, int use_overlapped); static PyObject * @@ -681,9 +681,9 @@ _winapi_ReadFile(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyOb { PyObject *return_value = NULL; static const char * const _keywords[] = {"handle", "size", "overlapped", NULL}; - static _PyArg_Parser _parser = {"" F_HANDLE "i|i:ReadFile", _keywords, 0}; + static _PyArg_Parser _parser = {"" F_HANDLE "k|i:ReadFile", _keywords, 0}; HANDLE handle; - int size; + DWORD size; int use_overlapped = 0; if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, @@ -941,4 +941,4 @@ _winapi_GetFileType(PyObject *module, PyObject *const *args, Py_ssize_t nargs, P exit: return return_value; } -/*[clinic end generated code: output=ec7f36eb7efc9d00 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=baaf3d379b91be0a input=a9049054013a1b77]*/ From webhook-mailer at python.org Wed Sep 5 02:29:47 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Wed, 05 Sep 2018 06:29:47 -0000 Subject: [Python-checkins] closes bpo-34581 : Conditionalize use of __pragma in Modules/socketmodule.c. (GH-9067) Message-ID: https://github.com/python/cpython/commit/874809ea389e6434787e773a6054a08e0b81f734 commit: 874809ea389e6434787e773a6054a08e0b81f734 branch: master author: Erik Janssens committer: Benjamin Peterson date: 2018-09-04T23:29:42-07:00 summary: closes bpo-34581 : Conditionalize use of __pragma in Modules/socketmodule.c. (GH-9067) files: A Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst M Modules/socketmodule.c diff --git a/Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst b/Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst new file mode 100644 index 000000000000..2dfa1aec9b86 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst @@ -0,0 +1 @@ +Guard MSVC-specific code in socketmodule.c with ``#ifdef _MSC_VER``. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 3a439c4bfadc..014f3ba8b5ea 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -559,15 +559,18 @@ select_error(void) # define SET_SOCK_ERROR(err) WSASetLastError(err) # define SOCK_TIMEOUT_ERR WSAEWOULDBLOCK # define SOCK_INPROGRESS_ERR WSAEWOULDBLOCK -# define SUPPRESS_DEPRECATED_CALL __pragma(warning(suppress: 4996)) #else # define GET_SOCK_ERROR errno # define SET_SOCK_ERROR(err) do { errno = err; } while (0) # define SOCK_TIMEOUT_ERR EWOULDBLOCK # define SOCK_INPROGRESS_ERR EINPROGRESS -# define SUPPRESS_DEPRECATED_CALL #endif +#ifdef _MSC_VER +# define SUPPRESS_DEPRECATED_CALL __pragma(warning(suppress: 4996)) +#else +# define SUPPRESS_DEPRECATED_CALL +#endif #ifdef MS_WINDOWS /* Does WSASocket() support the WSA_FLAG_NO_HANDLE_INHERIT flag? */ From webhook-mailer at python.org Wed Sep 5 02:45:07 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 05 Sep 2018 06:45:07 -0000 Subject: [Python-checkins] closes bpo-34581 : Conditionalize use of __pragma in Modules/socketmodule.c. (GH-9067) Message-ID: https://github.com/python/cpython/commit/5b17d7fccd8f0b4d5030b03924eb00904585ba31 commit: 5b17d7fccd8f0b4d5030b03924eb00904585ba31 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-04T23:45:03-07:00 summary: closes bpo-34581 : Conditionalize use of __pragma in Modules/socketmodule.c. (GH-9067) (cherry picked from commit 874809ea389e6434787e773a6054a08e0b81f734) Co-authored-by: Erik Janssens files: A Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst M Modules/socketmodule.c diff --git a/Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst b/Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst new file mode 100644 index 000000000000..2dfa1aec9b86 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2018-09-04-23-13-19.bpo-34581.lnbC0k.rst @@ -0,0 +1 @@ +Guard MSVC-specific code in socketmodule.c with ``#ifdef _MSC_VER``. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index ea01a286874b..f40bd89e577f 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -559,15 +559,18 @@ select_error(void) # define SET_SOCK_ERROR(err) WSASetLastError(err) # define SOCK_TIMEOUT_ERR WSAEWOULDBLOCK # define SOCK_INPROGRESS_ERR WSAEWOULDBLOCK -# define SUPPRESS_DEPRECATED_CALL __pragma(warning(suppress: 4996)) #else # define GET_SOCK_ERROR errno # define SET_SOCK_ERROR(err) do { errno = err; } while (0) # define SOCK_TIMEOUT_ERR EWOULDBLOCK # define SOCK_INPROGRESS_ERR EINPROGRESS -# define SUPPRESS_DEPRECATED_CALL #endif +#ifdef _MSC_VER +# define SUPPRESS_DEPRECATED_CALL __pragma(warning(suppress: 4996)) +#else +# define SUPPRESS_DEPRECATED_CALL +#endif #ifdef MS_WINDOWS /* Does WSASocket() support the WSA_FLAG_NO_HANDLE_INHERIT flag? */ From solipsis at pitrou.net Wed Sep 5 05:10:07 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 05 Sep 2018 09:10:07 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=-3 Message-ID: <20180905091007.1.2F0357B9DCAD4983@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [-7, 1, 0] memory blocks, sum=-6 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_fork leaked [-2, 2, -1] memory blocks, sum=-1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/refloglPX746', '--timeout', '7200'] From webhook-mailer at python.org Wed Sep 5 10:42:56 2018 From: webhook-mailer at python.org (Zachary Ware) Date: Wed, 05 Sep 2018 14:42:56 -0000 Subject: [Python-checkins] [3.6] bpo-34575: Build with only VS2015 on AppVeyor (GH-9066) Message-ID: https://github.com/python/cpython/commit/635461fca5e90c6e091f1e5b46adafc0d28bf0e2 commit: 635461fca5e90c6e091f1e5b46adafc0d28bf0e2 branch: 3.6 author: Zachary Ware committer: GitHub date: 2018-09-05T09:42:53-05:00 summary: [3.6] bpo-34575: Build with only VS2015 on AppVeyor (GH-9066) Remove VS2017 image from AppVeyor config. We're currently having caching issues between the two images, and VS2017 builds are tested by VSTS. Removing that one here will significantly decrease the time that AppVeyor builds take on this branch. files: M .github/appveyor.yml diff --git a/.github/appveyor.yml b/.github/appveyor.yml index 09eb7db0f0d4..3e0f588a2892 100644 --- a/.github/appveyor.yml +++ b/.github/appveyor.yml @@ -36,4 +36,3 @@ environment: HOST_PYTHON: C:\Python36\python.exe image: - Visual Studio 2015 - - Visual Studio 2017 From webhook-mailer at python.org Wed Sep 5 10:45:11 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 05 Sep 2018 14:45:11 -0000 Subject: [Python-checkins] bpo-34530: Fix distutils find_executable() (GH-9049) (GH-9057) Message-ID: https://github.com/python/cpython/commit/e2c1657dff86decf1e232b66e766d2e51381109c commit: e2c1657dff86decf1e232b66e766d2e51381109c branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-05T16:45:08+02:00 summary: bpo-34530: Fix distutils find_executable() (GH-9049) (GH-9057) distutils.spawn.find_executable() now falls back on os.defpath if the PATH environment variable is not set. (cherry picked from commit 39487196c87e28128ea907a0d9b8a88ba53f68d5) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst M Lib/distutils/spawn.py M Lib/distutils/tests/test_spawn.py diff --git a/Lib/distutils/spawn.py b/Lib/distutils/spawn.py index 5dd415a283d1..538768809327 100644 --- a/Lib/distutils/spawn.py +++ b/Lib/distutils/spawn.py @@ -173,7 +173,7 @@ def find_executable(executable, path=None): os.environ['PATH']. Returns the complete filename or None if not found. """ if path is None: - path = os.environ['PATH'] + path = os.environ.get('PATH', os.defpath) paths = path.split(os.pathsep) base, ext = os.path.splitext(executable) diff --git a/Lib/distutils/tests/test_spawn.py b/Lib/distutils/tests/test_spawn.py index 5edc24a3a104..0d455385d8ac 100644 --- a/Lib/distutils/tests/test_spawn.py +++ b/Lib/distutils/tests/test_spawn.py @@ -1,9 +1,13 @@ """Tests for distutils.spawn.""" -import unittest -import sys import os +import stat +import sys +import unittest +from unittest import mock from test.support import run_unittest, unix_shell +from test import support as test_support +from distutils.spawn import find_executable from distutils.spawn import _nt_quote_args from distutils.spawn import spawn from distutils.errors import DistutilsExecError @@ -51,6 +55,47 @@ def test_spawn(self): os.chmod(exe, 0o777) spawn([exe]) # should work without any error + def test_find_executable(self): + with test_support.temp_dir() as tmp_dir: + # use TESTFN to get a pseudo-unique filename + program_noeext = test_support.TESTFN + # Give the temporary program an ".exe" suffix for all. + # It's needed on Windows and not harmful on other platforms. + program = program_noeext + ".exe" + + filename = os.path.join(tmp_dir, program) + with open(filename, "wb"): + pass + os.chmod(filename, stat.S_IXUSR) + + # test path parameter + rv = find_executable(program, path=tmp_dir) + self.assertEqual(rv, filename) + + if sys.platform == 'win32': + # test without ".exe" extension + rv = find_executable(program_noeext, path=tmp_dir) + self.assertEqual(rv, filename) + + # test find in the current directory + with test_support.change_cwd(tmp_dir): + rv = find_executable(program) + self.assertEqual(rv, program) + + # test non-existent program + dont_exist_program = "dontexist_" + program + rv = find_executable(dont_exist_program , path=tmp_dir) + self.assertIsNone(rv) + + # test os.defpath: missing PATH environment variable + with test_support.EnvironmentVarGuard() as env: + with mock.patch('distutils.spawn.os.defpath', tmp_dir): + env.pop('PATH') + + rv = find_executable(program) + self.assertEqual(rv, filename) + + def test_suite(): return unittest.makeSuite(SpawnTestCase) diff --git a/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst new file mode 100644 index 000000000000..064de73c0ffe --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-03-23-23-32.bpo-34530.h_Xsu7.rst @@ -0,0 +1,2 @@ +``distutils.spawn.find_executable()`` now falls back on :data:`os.defpath` +if the ``PATH`` environment variable is not set. From webhook-mailer at python.org Wed Sep 5 10:46:01 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 05 Sep 2018 14:46:01 -0000 Subject: [Python-checkins] [3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8970) (GH-9061) Message-ID: https://github.com/python/cpython/commit/1a3eb125dc07a28a5af62446778ed7cca95ed3da commit: 1a3eb125dc07a28a5af62446778ed7cca95ed3da branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-05T16:45:57+02:00 summary: [3.7] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8970) (GH-9061) (cherry picked from commit 7d81e8f5995df6980a1a02923e224a481375f130) (cherry picked from commit 20a8392cec2967f15ae81633c1775645b3ca40da) Co-authored-by: Serhiy Storchaka files: M Lib/platform.py M Lib/test/test_platform.py diff --git a/Lib/platform.py b/Lib/platform.py index c8e0476bfb21..4205abde0388 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -136,6 +136,35 @@ # Constant used by test_platform to test linux_distribution(). _UNIXCONFDIR = '/etc' +# Helper for comparing two version number strings. +# Based on the description of the PHP's version_compare(): +# http://php.net/manual/en/function.version-compare.php + +_ver_stages = { + # any string not found in this dict, will get 0 assigned + 'dev': 10, + 'alpha': 20, 'a': 20, + 'beta': 30, 'b': 30, + 'c': 40, + 'RC': 50, 'rc': 50, + # number, will get 100 assigned + 'pl': 200, 'p': 200, +} + +_component_re = re.compile(r'([0-9]+|[._+-])') + +def _comparable_version(version): + result = [] + for v in _component_re.split(version): + if v not in '._+-': + try: + v = int(v, 10) + t = 100 + except ValueError: + t = _ver_stages.get(v, 0) + result.extend((t, v)) + return result + ### Platform specific APIs _libc_search = re.compile(b'(__libc_init)' @@ -159,7 +188,7 @@ def libc_ver(executable=sys.executable, lib='', version='', chunksize=16384): The file is read and scanned in chunks of chunksize bytes. """ - from distutils.version import LooseVersion as V + V = _comparable_version if hasattr(os.path, 'realpath'): # Python 2.2 introduced os.path.realpath(); it is used # here to work around problems with Cygwin not being diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py index afc4de326907..c8ba7ec35947 100644 --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -285,6 +285,42 @@ def test_libc_ver(self): self.assertEqual(platform.libc_ver(support.TESTFN), ('glibc', '1.23.4')) + @support.cpython_only + def test__comparable_version(self): + from platform import _comparable_version as V + self.assertEqual(V('1.2.3'), V('1.2.3')) + self.assertLess(V('1.2.3'), V('1.2.10')) + self.assertEqual(V('1.2.3.4'), V('1_2-3+4')) + self.assertLess(V('1.2spam'), V('1.2dev')) + self.assertLess(V('1.2dev'), V('1.2alpha')) + self.assertLess(V('1.2dev'), V('1.2a')) + self.assertLess(V('1.2alpha'), V('1.2beta')) + self.assertLess(V('1.2a'), V('1.2b')) + self.assertLess(V('1.2beta'), V('1.2c')) + self.assertLess(V('1.2b'), V('1.2c')) + self.assertLess(V('1.2c'), V('1.2RC')) + self.assertLess(V('1.2c'), V('1.2rc')) + self.assertLess(V('1.2RC'), V('1.2.0')) + self.assertLess(V('1.2rc'), V('1.2.0')) + self.assertLess(V('1.2.0'), V('1.2pl')) + self.assertLess(V('1.2.0'), V('1.2p')) + + self.assertLess(V('1.5.1'), V('1.5.2b2')) + self.assertLess(V('3.10a'), V('161')) + self.assertEqual(V('8.02'), V('8.02')) + self.assertLess(V('3.4j'), V('1996.07.12')) + self.assertLess(V('3.1.1.6'), V('3.2.pl0')) + self.assertLess(V('2g6'), V('11g')) + self.assertLess(V('0.9'), V('2.2')) + self.assertLess(V('1.2'), V('1.2.1')) + self.assertLess(V('1.1'), V('1.2.2')) + self.assertLess(V('1.1'), V('1.2')) + self.assertLess(V('1.2.1'), V('1.2.2')) + self.assertLess(V('1.2'), V('1.2.2')) + self.assertLess(V('0.4'), V('0.4.0')) + self.assertLess(V('1.13++'), V('5.5.kw')) + self.assertLess(V('0.960923'), V('2.2beta29')) + def test_parse_release_file(self): for input, output in ( From webhook-mailer at python.org Wed Sep 5 10:46:29 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 05 Sep 2018 14:46:29 -0000 Subject: [Python-checkins] bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8952) Message-ID: https://github.com/python/cpython/commit/9734024ec65311e33936faa83fb1cb249ef0de9d commit: 9734024ec65311e33936faa83fb1cb249ef0de9d branch: 2.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-05T16:46:26+02:00 summary: bpo-26544: Get rid of dependence from distutils in platform. (GH-8356) (GH-8952) (cherry picked from commit 7d81e8f5995df6980a1a02923e224a481375f130) Co-authored-by: Serhiy Storchaka files: M Lib/platform.py diff --git a/Lib/platform.py b/Lib/platform.py index 44a612b28fce..62a5476a8f4d 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -132,6 +132,35 @@ # Standard Unix uses /dev/null DEV_NULL = '/dev/null' +# Helper for comparing two version number strings. +# Based on the description of the PHP's version_compare(): +# http://php.net/manual/en/function.version-compare.php + +_ver_stages = { + # any string not found in this dict, will get 0 assigned + 'dev': 10, + 'alpha': 20, 'a': 20, + 'beta': 30, 'b': 30, + 'c': 40, + 'RC': 50, 'rc': 50, + # number, will get 100 assigned + 'pl': 200, 'p': 200, +} + +_component_re = re.compile(r'([0-9]+|[._+-])') + +def _comparable_version(version): + result = [] + for v in _component_re.split(version): + if v not in '._+-': + try: + v = int(v, 10) + t = 100 + except ValueError: + t = _ver_stages.get(v, 0) + result.extend((t, v)) + return result + ### Platform specific APIs _libc_search = re.compile(r'(__libc_init)' @@ -155,7 +184,7 @@ def libc_ver(executable=sys.executable,lib='',version='', chunksize=2048): The file is read and scanned in chunks of chunksize bytes. """ - from distutils.version import LooseVersion as V + V = _comparable_version if hasattr(os.path, 'realpath'): # Python 2.2 introduced os.path.realpath(); it is used # here to work around problems with Cygwin not being From solipsis at pitrou.net Thu Sep 6 05:10:00 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 06 Sep 2018 09:10:00 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=7 Message-ID: <20180906091000.1.41C98DCDEDCBC8BC@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [-7, 8, 0] memory blocks, sum=1 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_spawn leaked [0, 2, 0] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogqXbjzO', '--timeout', '7200'] From webhook-mailer at python.org Thu Sep 6 07:34:32 2018 From: webhook-mailer at python.org (Tal Einat) Date: Thu, 06 Sep 2018 11:34:32 -0000 Subject: [Python-checkins] bpo-30977: make uuid.UUID use __slots__ (GH-9078) Message-ID: https://github.com/python/cpython/commit/3e2b29dccc3ca9fbc418bfa312ad655782e250f2 commit: 3e2b29dccc3ca9fbc418bfa312ad655782e250f2 branch: master author: Tal Einat committer: GitHub date: 2018-09-06T14:34:25+03:00 summary: bpo-30977: make uuid.UUID use __slots__ (GH-9078) Co-Authored-By: Wouter Bolsterlee. files: A Misc/NEWS.d/next/Library/2018-09-06-10-07-46.bpo-30977.bP661V.rst M Lib/test/test_uuid.py M Lib/uuid.py diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py index 7af1d7aec797..9ec59d519573 100644 --- a/Lib/test/test_uuid.py +++ b/Lib/test/test_uuid.py @@ -4,8 +4,10 @@ import contextlib import io import os +import pickle import shutil import subprocess +import sys py_uuid = support.import_fresh_module('uuid', blocked=['_uuid']) c_uuid = support.import_fresh_module('uuid', fresh=['_uuid']) @@ -311,6 +313,64 @@ def test_getnode(self): node2 = self.uuid.getnode() self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2)) + def _setup_for_pickle(self): + orig_uuid = sys.modules.get('uuid') + sys.modules['uuid'] = self.uuid + + def restore_uuid_module(): + if orig_uuid is not None: + sys.modules['uuid'] = orig_uuid + else: + del sys.modules['uuid'] + self.addCleanup(restore_uuid_module) + + def test_pickle_roundtrip(self): + self._setup_for_pickle() + + u = self.uuid.UUID('12345678123456781234567812345678') + self.assertEqual(u, pickle.loads(pickle.dumps(u))) + + def test_unpickle_previous_python_versions(self): + self._setup_for_pickle() + + u = self.uuid.UUID('12345678123456781234567812345678') + + # Python 2.7 protocol 0-2 pickles of u + py27_pickles = [ + b'ccopy_reg\n_reconstructor\np0\n(cuuid\nUUID\np1\nc__builtin__\nob' + b'ject\np2\nNtp3\nRp4\n(dp5\nS\'int\'\np6\nL24197857161011715162171' + b'839636988778104L\nsb.', + b'ccopy_reg\n_reconstructor\nq\x00(cuuid\nUUID\nq\x01c__builtin__\n' + b'object\nq\x02Ntq\x03Rq\x04}q\x05U\x03intq\x06L2419785716101171516' + b'2171839636988778104L\nsb.', + b'\x80\x02cuuid\nUUID\nq\x00)\x81q\x01}q\x02U\x03intq\x03\x8a\x10xV' + b'4\x12xV4\x12xV4\x12xV4\x12sb.', + ] + # Python 3.6 protocol 0-4 pickles of u + py36_pickles = [ + b'ccopy_reg\n_reconstructor\np0\n(cuuid\nUUID\np1\nc__builtin__\nob' + b'ject\np2\nNtp3\nRp4\n(dp5\nVint\np6\nL241978571610117151621718396' + b'36988778104L\nsb.', + b'ccopy_reg\n_reconstructor\nq\x00(cuuid\nUUID\nq\x01c__builtin__\n' + b'object\nq\x02Ntq\x03Rq\x04}q\x05X\x03\x00\x00\x00intq\x06L2419785' + b'7161011715162171839636988778104L\nsb.', + b'\x80\x02cuuid\nUUID\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00intq' + b'\x03\x8a\x10xV4\x12xV4\x12xV4\x12xV4\x12sb.', + b'\x80\x03cuuid\nUUID\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00intq' + b'\x03\x8a\x10xV4\x12xV4\x12xV4\x12xV4\x12sb.', + b'\x80\x04\x950\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c\x04' + b'UUID\x94\x93\x94)\x81\x94}\x94\x8c\x03int\x94\x8a\x10xV4\x12xV4' + b'\x12xV4\x12xV4\x12sb.', + ] + + for pickled in py27_pickles + py36_pickles: + unpickled = pickle.loads(pickled) + self.assertEqual(unpickled, u) + # is_safe was added in 3.7. When unpickling values from older + # versions, is_safe will be missing, so it should be set to + # SafeUUID.unknown. + self.assertEqual(unpickled.is_safe, self.uuid.SafeUUID.unknown) + # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers # need not necessarily be 48 bits (e.g., EUI-64). def test_uuid1_eui64(self): diff --git a/Lib/uuid.py b/Lib/uuid.py index 66383218e70c..515388221510 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -118,6 +118,8 @@ class UUID: uuid_generate_time_safe(3). """ + __slots__ = ('int', 'is_safe') + def __init__(self, hex=None, bytes=None, bytes_le=None, fields=None, int=None, version=None, *, is_safe=SafeUUID.unknown): @@ -201,8 +203,30 @@ def __init__(self, hex=None, bytes=None, bytes_le=None, fields=None, # Set the version number. int &= ~(0xf000 << 64) int |= version << 76 - self.__dict__['int'] = int - self.__dict__['is_safe'] = is_safe + object.__setattr__(self, 'int', int) + object.__setattr__(self, 'is_safe', is_safe) + + def __getstate__(self): + d = {attr: getattr(self, attr) for attr in self.__slots__} + # is_safe is a SafeUUID instance. Return just its value, so that + # it can be unpickled in older Python versions without SafeUUID. + d['is_safe'] = d['is_safe'].value + return d + + def __setstate__(self, state): + # is_safe was added in 3.7 + state.setdefault('is_safe', SafeUUID.unknown.value) + + for attr in self.__slots__: + value = state[attr] + + # for is_safe, restore the SafeUUID from the stored value + if attr == 'is_safe': + try: + value = SafeUUID(value) + except ValueError: + value = SafeUUID.unknown + object.__setattr__(self, attr, value) def __eq__(self, other): if isinstance(other, UUID): diff --git a/Misc/NEWS.d/next/Library/2018-09-06-10-07-46.bpo-30977.bP661V.rst b/Misc/NEWS.d/next/Library/2018-09-06-10-07-46.bpo-30977.bP661V.rst new file mode 100644 index 000000000000..3d547c06beb5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-06-10-07-46.bpo-30977.bP661V.rst @@ -0,0 +1,2 @@ +Make uuid.UUID use ``__slots__`` to reduce its memory footprint. Based on +original patch by Wouter Bolsterlee. From webhook-mailer at python.org Thu Sep 6 09:13:28 2018 From: webhook-mailer at python.org (Christian Heimes) Date: Thu, 06 Sep 2018 13:13:28 -0000 Subject: [Python-checkins] bpo-34542: Update test certs and keys (GH-8997) (GH-9007) Message-ID: https://github.com/python/cpython/commit/2d3f2dc9f8376415a31a6de260ccbe6a86f2816d commit: 2d3f2dc9f8376415a31a6de260ccbe6a86f2816d branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Christian Heimes date: 2018-09-06T15:13:24+02:00 summary: bpo-34542: Update test certs and keys (GH-8997) (GH-9007) Update all test certs and keys to use future proof crypto settings: * 3072 bit RSA keys * SHA-256 signature Signed-off-by: Christian Heimes (cherry picked from commit e6dac0077996b1e1f886f036d6f2606237fa4c85) files: A Misc/NEWS.d/next/Tests/2018-08-29-16-30-52.bpo-34542.9stVAW.rst M Lib/test/allsans.pem M Lib/test/capath/b1930218.0 M Lib/test/capath/ceff1710.0 M Lib/test/idnsans.pem M Lib/test/keycert.passwd.pem M Lib/test/keycert.pem M Lib/test/keycert2.pem M Lib/test/keycert3.pem M Lib/test/keycert4.pem M Lib/test/keycertecc.pem M Lib/test/make_ssl_certs.py M Lib/test/pycacert.pem M Lib/test/pycakey.pem M Lib/test/revocation.crl M Lib/test/ssl_cert.pem M Lib/test/ssl_key.passwd.pem M Lib/test/ssl_key.pem M Lib/test/test_asyncio/utils.py M Lib/test/test_ssl.py diff --git a/Lib/test/allsans.pem b/Lib/test/allsans.pem index bf59f30abaad..6eebde7a57f1 100644 --- a/Lib/test/allsans.pem +++ b/Lib/test/allsans.pem @@ -1,64 +1,81 @@ -----BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQD59JyhPgYe7nhZ -Z2IGhaklNgtRkD+5BVs7lEWovYRBlXpPA6PuaHat25rI8EGYHmlufPherg2Qu6sC -GmZZKo7TgjlmDcwVS4hkebtFH7OZy5Il7Y2ZIdiK7Xp9Z0EPoqwacYowB0a8WhZY -I2Vm4EzCNKl6/htkwjgn2JXGizxvGt/1kNqP/GBAX+vjgeahOsn8jVh96KpFHJbS -g83cX4t8M7FJv7yNoDLvORHnvKCOXbQmr6ZMGcZN8PwS8awQ31khZTpEx+hCe+Pi -GzeOlxpZimXWDAGWA4tZ58Ka/QvO7VQbD5Ci166ODvvs+tEXfBUExtPcS+02IBJV -tzhBna9VAgMBAAECggEAPar9DccIqY76QEyCYcuOPLEFv9zP6+0HYj6lpQkE3U1s -vJvQURyS0zgQCy1Dca1nI6xPdsSIckHq4fzzbWJTlJlXYfdbd5GIGAn0iwxUOkiA -ST0/px0zmKsYgmH8KkhfH7MNfeX9rLCpPJuXA/eo2G03tzGEPqqwQhxsb2ygv2Qs -M7OqJz6RJu87K1Y+psWIv9+VhNVja0kvsg52QMK9mtp8layb54qLI5R5e09sIudq -RHegtnSOBo9kt32H9vWUFaF5PpYt4yks4KYI4ulKGWJGXHMDW4uHUaE/tjNQuYAX -DuDvjN+ECSJvigiUbu2k0xB2KYIb1fpcxlz/YBdADQKBgQD/Z2VtBUjOFnJKz00f -xN0akp7XPgd1yCb1/wZq9PQiGvzIAMDIplioTvjOjhOzPJaWD0GICNeypzQ48+0P -UsPIKbazpIZN6bZncr65plSpg0KANq46hbkPHOo8PHDa7yoxBUSPr8F7P1OCRkn6 -+QdgcnrAly7yfqO2ahAWOX7iCwKBgQD6ifXSCKfRF1GUb3Ws7S1rLxeBasWq+EmC -sUnck0S+AyaMkN+kZ5zejbN+NDuUMQ7+3wUIheTclUhzR0LP3+r5jjHsimJuvOml -wuV37F+Om5lD/Xx27NfbtRKn/bK6o0zDL8JB2eFB0N7Fh7hRYoUMdrpQs5sU91IC -pNYlAcLwHwKBgGvLK9eTf2LbvmksjRR3dgodD8UwfN2NGESC2iaSM+ehFEclajhF -XO3MRt6GwHHJhJTY44OSl9bjEvtmmAr7l34HfQDc04JWvZFzsGOSe/D/YTXT3jz8 -61ohjgrWR5tfjaMa4hDy0Oo/k/NLzzWJnT9rkbtvE3VtVZNLuHZo1dB5AoGBAMHO -wStV6MO1nzUNN+Gqo8zbY/qIJxsH8I26KaIJBk9azpJEa8yZHl+HDEffjgsoHCqL -STB7qzv7+0y53nRCClo8ZmBN+LEjUDcbWjl3z7/YnCpdR9ATjTP3kdQETCNWucXw -Bvy72CX6tqnlQG8soDGxEpXlKl2AqJ9E9icwgqUPAoGAL6xTDdgcYTbk9wxCd41l -NhHTSvLrGXLAzv61PCnlOJEJbuuezb2VW0ibsud5CA4Mi0tf9ET790XSOFd5nCjQ -6rr06AkjQsoFvjL1dO9EzVFPW0JrZ3C9y8ZOjdeAfPEmFL2T6VqmQ+IcCUNhSr39 -NBdKrboEFfnKanfbstekhAs= +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQCg/pM6dP7BTFNc +qe6wIJIBB7HjwL42bp0vjcCVl4Z3MRWFswYpfxy+o+8+PguMp4K6zndA5fwNkK/H +3HmtanncUfPqnV0usN0NHQGh/f9xRoNmB1q2L7kTuO99o0KLQgvonRT2snf8rq9n +tPRzhHUGYhog7zzNxetYV309PHpPr19BcKepDtM5RMk2aBnoN5vtItorjXiDosFm +6o5wQHrcupcVydszba6P75BEbc1XIWvq2Fv8muaw4pCe81QYINyLqgcPNO/nF3Os +5EI4HKjCNRSCOhOcWqYctXLXN9lBdMBBvQc3zDmYzh1eIZewzZXPVEQT33xPkhxz +HNmhcIctpWX4LTRF6FulkcbeuZDga3gkZYJf/M6IpU1WYXr6q8sNxbgmRRX/NuHo +V9oDwBzLG07rKUiqRHfjGqoCRmmVeVYpryvXUNjHGH0nlVzz/8lTUxAnJorO3Fdc +I+6zKLUPICdAlvz51AH6yopgPFhrdgA0pVzPO6L5G8SRQCxKhAUCAwEAAQKCAYAa +2jtOTcNMFGH3G7TfFZ+kolbuaPCQ/aQkEV2k1dAswzgWw8RsWXI+7fLyi8C7Zhks +9VD4tyNyU8at7D0zSoYm1Fh9sl+fcQp9rG/gSBA6IYu7EdD0gEM7YeY4K2nm9k4s +Lz8W4q+WqsBA6PK47cfjF6vKAH1AyRk28+jEtPiln9egf5zHWtyqOanh9D0V+Wh9 +hgmjqAYI1rWxZ7/4Qxj7Bfg7Px7blhi+kzOZ5kKQnNd2JT46hM+jgzah/G3zVE+R +FFW6ksmJgZ+dCuSbE7HEJmKms1CWq/1Cll0A3uy4JTDZOrK4KcZQ9UjjWJWvlXQm +uNXSSAp1k287DLVUm9c22SDeXpb9PyKmzyvJvVmMqqBx6QzHZ/L7WPzpUWAoLcU+ +ZHT7vggDymkIO+fcRbUzv8s5R7RnLbcBga51/5OCUvAWDoJXNw0qwYZOIbfTnQgs +8xbCmbMzllyYM/dK3GxQAwfn8Hzk+DbS/NObMjHLCWLfYeUvutXJSNly6Ny+ZcEC +gcEAzo5Y1UFOfBX4MZLIZ69LfgaXj9URobMwqlEwKil8pWQMa951ga3moLt91nOe +SAQz3meFTBX/VAb2ZHLeIf3FoNkiIx48PkxsR/hhLHpvl26zEg3yXs3tv0IFBx2R +EEnLNpQaAQFR9S1yDOaG2rsb17ZDKyp9isDpAENHAmEnT/XJn+Dc0SOH1EVDjUeM +JqToAF/fjIx/RF4oUJCAgOPBMlRy5ywLQk8uDi6ft0NCzzCi0eCuk1Ty3KzWFGwx +7cYRAoHBAMeIPCzHG3No4JGUFunslVwo5TuC7maO6qYKbq0OyvwWfL4b7gjrMBR9 +d5WyZlp/Vf40O463dg8x8qPNOFWp49f3hxTvvfnt2/m3+CQuDOLfqBbHufZApP1J +U9MubUNnDFHHeJ9l0tg2nhiLw24GHeMARZhA/BimMQPY0OpZPpLVxAUArM2EB7hI +glQpYCtdXhqwl1pl0u3TZ08y3BXYNg9BycdpGRMWSsAwsApJRgNuI/dfDKu0uMYF +/pUhXVPatQKBwGgLpAun3dT7bA3sli5ESo6s22OEPGFrVbQ1OUHDrBnTj742TJKJ ++oY0a2q+ypgUJdx94NM2sWquJybqBaKxpf8j4OI3tLjc3h5SqwAwnE13YZRSmifP +K1cP9mBjMFM4GLjhWUfwVkxeG/kLlhpP7fJ2yNbRjHN8QOH1AavdLGRGts1mA1UF +xMHUMfbUd3Bv2L13ja/KhcD2fPA4GcLS9tpXV5nCwdkg8V4LdkBmDR04rotx1f44 +6Czokt2usmfHQQKBwFkufxbUd2SB/72Rnxw27hse/DY5My0Lu70y9HzNG9TIiEDA +YwgBdp/x5D04W58fQuQ3nFcRkOcBwB2OYBuJr5ibvfiRnyvSMHvQykwBeSj+Jjbo +VinGgvfiimDdY2C48jyrFzLHZBHXd5oo/dRzT3Bicri2cvbhcQ7zHY1hDiK7AL3r +q1DALmMjpXzQcXdwZ9suCrgQwtIhpw8zAEOTO7ZeBT3nr5lkYUy9djFixrRJyjGK +fjNQtzVrAHrPStNr8QKBwQDCC0zhsCnTv4sAJmW7LL6Ayd5rbWhUZ6px1xY0yHMA +hehj+xbaiC6cfVr5Rg0ncvaa8AExu4kXpVsupTyNwvC4NgzLHtfBw6WUdOnd1awE +kSrDtDReBt2wByAcQwttQsrJ1/Pt6zcNJJI4Z9s8G4NTcQWJwUhU20N55JQKR//l +OQJqhq9NVhte/ctDjVwOHs/OhDNvxsAWxdjnf/O2up0os+M2bFkmHuaVW0vQbqTQ +mw7Vbzk2Ff5oT6E3kbC8Ur4= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIIGMDCCBRigAwIBAgIJAJYf8T95ptq5MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV +MIIHMDCCBZigAwIBAgIJALVVA6v9zJS5MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnMwHhcNMTgwMTE5 -MTkwOTA3WhcNMjgwMTE3MTkwOTA3WjBdMQswCQYDVQQGEwJYWTEXMBUGA1UEBwwO +IFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnMwHhcNMTgwODI5 +MTQyMzE3WhcNMjgwODI2MTQyMzE3WjBdMQswCQYDVQQGEwJYWTEXMBUGA1UEBwwO Q2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5dGhvbiBTb2Z0d2FyZSBGb3VuZGF0 -aW9uMRAwDgYDVQQDDAdhbGxzYW5zMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB -CgKCAQEA+fScoT4GHu54WWdiBoWpJTYLUZA/uQVbO5RFqL2EQZV6TwOj7mh2rdua -yPBBmB5pbnz4Xq4NkLurAhpmWSqO04I5Zg3MFUuIZHm7RR+zmcuSJe2NmSHYiu16 -fWdBD6KsGnGKMAdGvFoWWCNlZuBMwjSpev4bZMI4J9iVxos8bxrf9ZDaj/xgQF/r -44HmoTrJ/I1YfeiqRRyW0oPN3F+LfDOxSb+8jaAy7zkR57ygjl20Jq+mTBnGTfD8 -EvGsEN9ZIWU6RMfoQnvj4hs3jpcaWYpl1gwBlgOLWefCmv0Lzu1UGw+Qoteujg77 -7PrRF3wVBMbT3EvtNiASVbc4QZ2vVQIDAQABo4IC8TCCAu0wggEwBgNVHREEggEn -MIIBI4IHYWxsc2Fuc6AeBgMqAwSgFwwVc29tZSBvdGhlciBpZGVudGlmaWVyoDUG -BisGAQUCAqArMCmgEBsOS0VSQkVST1MuUkVBTE2hFTAToAMCAQGhDDAKGwh1c2Vy -bmFtZYEQdXNlckBleGFtcGxlLm9yZ4IPd3d3LmV4YW1wbGUub3JnpGcwZTELMAkG -A1UEBhMCWFkxFzAVBgNVBAcMDkNhc3RsZSBBbnRocmF4MSMwIQYDVQQKDBpQeXRo -b24gU29mdHdhcmUgRm91bmRhdGlvbjEYMBYGA1UEAwwPZGlybmFtZSBleGFtcGxl -hhdodHRwczovL3d3dy5weXRob24ub3JnL4cEfwAAAYcQAAAAAAAAAAAAAAAAAAAA -AYgEKgMEBTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG -AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFH9ye3+WhBnHqNhtFu059bzY -SWM8MIGPBgNVHSMEgYcwgYSAFH9ye3+WhBnHqNhtFu059bzYSWM8oWGkXzBdMQsw -CQYDVQQGEwJYWTEXMBUGA1UEBwwOQ2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5 -dGhvbiBTb2Z0d2FyZSBGb3VuZGF0aW9uMRAwDgYDVQQDDAdhbGxzYW5zggkAlh/x -P3mm2rkwgYMGCCsGAQUFBwEBBHcwdTA8BggrBgEFBQcwAoYwaHR0cDovL3Rlc3Rj -YS5weXRob250ZXN0Lm5ldC90ZXN0Y2EvcHljYWNlcnQuY2VyMDUGCCsGAQUFBzAB -hilodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3RjYS9vY3NwLzBDBgNV -HR8EPDA6MDigNqA0hjJodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3Rj -YS9yZXZvY2F0aW9uLmNybDANBgkqhkiG9w0BAQsFAAOCAQEAYwYJcerUPvnsP7e2 -HGp/It0OZ8Cvpt8Qf7A+NSPvJqkyKakl8zK/50iq/qQKH09CnfEae4rfXLdlYsvV -2PZYK0LDWnyTcHSJWAVJjlSFIFt3ig9FdHv9GYtSWWod66cZ0sEZOoF2IHZUGby+ -Qa+JQpmv5jEuGIZzjcsh6hSOou8ph7LsCsRdVlQqk8rM97vB7DAgh01vedlbolsq -JxsuPRydNFV/eWq3AgAWgZL3LdYYIAgaVOTnnd3xARw8DlT1q6+Lzc71GBXrRZYh -qgd+xC/K1812gMPImTX02bxpkhCuIdVd7cztWi8sdQmSgDEFdYMXo4NzlFTK8dlC -Y4wa3Q== +aW9uMRAwDgYDVQQDDAdhbGxzYW5zMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIB +igKCAYEAoP6TOnT+wUxTXKnusCCSAQex48C+Nm6dL43AlZeGdzEVhbMGKX8cvqPv +Pj4LjKeCus53QOX8DZCvx9x5rWp53FHz6p1dLrDdDR0Bof3/cUaDZgdati+5E7jv +faNCi0IL6J0U9rJ3/K6vZ7T0c4R1BmIaIO88zcXrWFd9PTx6T69fQXCnqQ7TOUTJ +NmgZ6Deb7SLaK414g6LBZuqOcEB63LqXFcnbM22uj++QRG3NVyFr6thb/JrmsOKQ +nvNUGCDci6oHDzTv5xdzrORCOByowjUUgjoTnFqmHLVy1zfZQXTAQb0HN8w5mM4d +XiGXsM2Vz1REE998T5IccxzZoXCHLaVl+C00RehbpZHG3rmQ4Gt4JGWCX/zOiKVN +VmF6+qvLDcW4JkUV/zbh6FfaA8AcyxtO6ylIqkR34xqqAkZplXlWKa8r11DYxxh9 +J5Vc8//JU1MQJyaKztxXXCPusyi1DyAnQJb8+dQB+sqKYDxYa3YANKVczzui+RvE +kUAsSoQFAgMBAAGjggLxMIIC7TCCATAGA1UdEQSCAScwggEjggdhbGxzYW5zoB4G +AyoDBKAXDBVzb21lIG90aGVyIGlkZW50aWZpZXKgNQYGKwYBBQICoCswKaAQGw5L +RVJCRVJPUy5SRUFMTaEVMBOgAwIBAaEMMAobCHVzZXJuYW1lgRB1c2VyQGV4YW1w +bGUub3Jngg93d3cuZXhhbXBsZS5vcmekZzBlMQswCQYDVQQGEwJYWTEXMBUGA1UE +BwwOQ2FzdGxlIEFudGhyYXgxIzAhBgNVBAoMGlB5dGhvbiBTb2Z0d2FyZSBGb3Vu +ZGF0aW9uMRgwFgYDVQQDDA9kaXJuYW1lIGV4YW1wbGWGF2h0dHBzOi8vd3d3LnB5 +dGhvbi5vcmcvhwR/AAABhxAAAAAAAAAAAAAAAAAAAAABiAQqAwQFMA4GA1UdDwEB +/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/ +BAIwADAdBgNVHQ4EFgQUoLHAHNTWrHkSCUYkhn5NH0S40CAwgY8GA1UdIwSBhzCB +hIAUoLHAHNTWrHkSCUYkhn5NH0S40CChYaRfMF0xCzAJBgNVBAYTAlhZMRcwFQYD +VQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZv +dW5kYXRpb24xEDAOBgNVBAMMB2FsbHNhbnOCCQC1VQOr/cyUuTCBgwYIKwYBBQUH +AQEEdzB1MDwGCCsGAQUFBzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0 +L3Rlc3RjYS9weWNhY2VydC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2Eu +cHl0aG9udGVzdC5uZXQvdGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0 +dHA6Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3Js +MA0GCSqGSIb3DQEBCwUAA4IBgQAeKJKycO2DES98gyR2e/GzPYEw87cCS0cEpiiP +3CEUgzfEbF0X89GDKEey4H3Irvosbvt2hEcf2RNpahLUL/fUv53bDmHNmL8qJg5E +UJVMOHvOpSOjqoqeRuSyG0GnnAuUwcxdrZY6UzLdslhuq9F8UjgHr6KSMx56G9uK +LmTy5njMab0in2xL/YRX/0nogK3BHqpUHrfCdEYZkciRxtAa+OPpWn4dcZi+Fpf7 +ZYSgPLNt+djtFDMIAk5Bo+XDaQdW3dhF0w44enrGAOV0xPE+/jOuenNhKBafjuNb +lkeSr45+QZsi1rd18ny8z3uuaGqIAziFgmllZOH2D8giTn6+5jZcCNZCoGKUkPI9 +l/GMWwxg4HQYYlZcsZzTCem9Rb2XcrasAbmhFapMtR+QAwSed5vKE7ZdtQhj74kB +7Q0E7Lkgpp6BaObb2As8/f0K/UlSVSvrYk+i3JT9wK/qqkRGxsTFEF7N9t0rKu8y +4JdQDtZCI552MsFvYW6m+IOYgxg= -----END CERTIFICATE----- diff --git a/Lib/test/capath/b1930218.0 b/Lib/test/capath/b1930218.0 index 07556ff9071a..730e7fd911a5 100644 --- a/Lib/test/capath/b1930218.0 +++ b/Lib/test/capath/b1930218.0 @@ -1,21 +1,26 @@ -----BEGIN CERTIFICATE----- -MIIDbTCCAlWgAwIBAgIJAILtv0HIgJGbMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIEbTCCAtWgAwIBAgIJAMstgJlaaVJbMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yODAxMTcx -OTA5MDZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg -Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBAMMYaWvJRymYjrFWwi76Dl68I4CzB2Ik0kJb -8Uq/qcghdcjj5iwfhzxufBvtOTKVt0CyYEjDmhYI/m1niDQ7d3dwHHBa0R9fBCFU -uQzjQYUdWO4v7fMO79gjofpz+0wo4OXmTQsCUkmGx75+veZWdotwjgqPBjMgHXtb -qtDFG6ubzFQJPL/kQGbx+9b3Fp3EGdTD8v8HvG9aniUbAkql7EKWOnDSbJkrzr7o -0gHv1bqwz5Q+gtAB1ktxgAMKEkWGeYHYS9LotbcsbJpMihAQ5OT1386EkcrRRuCE -cxdm22lDeICDvhRN8T4a1mz13kXzOa+R1T1URL9BzHNoGvwk25ECAwEAAaNQME4w -HQYDVR0OBBYEFJrPz27rcT3bPPGuiGtWcgPLCKdIMB8GA1UdIwQYMBaAFJrPz27r -cT3bPPGuiGtWcgPLCKdIMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB -ABAlyNwMVVzLg25573fsDY4MBsFLDNb3dVIhuBdKOIids3jEQvu4fBQ4EPus2hEA -W0KHXkWfbU5CpJoYBjkPRaaWiTLWWbPTjuOVtsSiS3QvZ8H7u/lybzdK5/RIM3Hf -uPXmQT/V1S8mCfgOkv9w6varWPuQBNZDLo+x+waradDcqPhbB/LUZh9j+F3BnkFE -u8nofeBG5KfIMl8xYuUcXIndt6JPng0TuF+xhFNMH84Z4QEAXr9BVZSppRPb8lnz -1k65nZ25CtmyGG18sfeWqr32+ZUPSm48fEZb39R47Jrc4uMB5oh3OZOcuipj+SVL -T6wIeTnGe98HNbrAwlC/WoE= +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA4MjYx +NDIzMTZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg +Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCAaIwDQYJKoZI +hvcNAQEBBQADggGPADCCAYoCggGBAJftVUG6NheV23Ec0+FhrFhz48aWzysfuAj1 +nUtLxzD2uAuzUnKgu8lNO47fIo4BV4HJknPMAMbscLA6F0DB3/KMNkzEp4HntiRo +4qB+NQcvoFv5RUb3HvBGEf7KGjxQ8SapX5winPhB4d9PEpUZL1yQARdufj59z+kJ +ryX4+EJ3LW1fNvJ4Hn1Kh2hjbAZxG436Jf7U0/WlF7Hv6hfLVMgnmYDLPEXxLFIc +3R9RRSBQHl6rV3MbQXiW3oSket2PMIU2WHl2oNJhyBuplJljxu74FL+0UlYxl/rr +rFOelc5MxFpKt8oDJ1s1V84C3OzKafiKWjnLFiAVAyRhbPR6/LZI5VkQXEnQI5/7 +cV466WifNHKAJ7Y/TLHZ22N/Z2hKbhH46MD0WhY5Uwto3nf6Ref4kc14zSiUl3FU ++8/wN97JJsXcG56JbQmsyERxy23xlzHVTCAzv3VKoODcaRHtKrRkEBEwiw6wpxDY +isWqG8gmiiXnZp+lahovfF+DxnhPHwIDAQABo1AwTjAdBgNVHQ4EFgQU3b/K2ubR +NLo3dSHKb5oIKPI1tkgwHwYDVR0jBBgwFoAU3b/K2ubRNLo3dSHKb5oIKPI1tkgw +DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAYEAM2pU02vA1wFfnfQFwZNm +kFDQtxjpsB5KoLbadpOvhNutFVQxFRPk3n5ODNUJHDQ1tuVM1m9lfTJf6/ypawf3 +SYLlgX4HgJpj+CzDQLyP1CraPtHuCLdNp4TK9D+hmEW+sQVp59/Xmasb7oswzPf8 +59QLF66Xv+R7/Q+ntIV541niFoe/HylFLCOTdr7Ahx3e7CtCauW7yPQKSggKjFzY +fU3RuL/V9yntktGUBOg1Bld/LCOXh6U1jSbTGkfyFtfZxtQfI0PTJpk5yiD0cSNv +DEp2dvd2H7P+v0ew/CpWgeHS3e4I2PT/WtwlYYqRArmGHPJQc3YlNfy2JSYVy+vE +K2EMHOfuLxeb7PDUoYTn0q/e5BskFKcBh+OrKVhGoNnACuCN11nTG/hUID54paXI +T4sDxJaf7PtHz3YtjWU0J7/6rgEFivOSCt2JbJehx+dgUees60t9LLhlyf5dakhV +juTH+WpA4bhkRem1WSmlX899WH1keeWkCawedmU9lMRo -----END CERTIFICATE----- diff --git a/Lib/test/capath/ceff1710.0 b/Lib/test/capath/ceff1710.0 index 07556ff9071a..730e7fd911a5 100644 --- a/Lib/test/capath/ceff1710.0 +++ b/Lib/test/capath/ceff1710.0 @@ -1,21 +1,26 @@ -----BEGIN CERTIFICATE----- -MIIDbTCCAlWgAwIBAgIJAILtv0HIgJGbMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIEbTCCAtWgAwIBAgIJAMstgJlaaVJbMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yODAxMTcx -OTA5MDZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg -Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBAMMYaWvJRymYjrFWwi76Dl68I4CzB2Ik0kJb -8Uq/qcghdcjj5iwfhzxufBvtOTKVt0CyYEjDmhYI/m1niDQ7d3dwHHBa0R9fBCFU -uQzjQYUdWO4v7fMO79gjofpz+0wo4OXmTQsCUkmGx75+veZWdotwjgqPBjMgHXtb -qtDFG6ubzFQJPL/kQGbx+9b3Fp3EGdTD8v8HvG9aniUbAkql7EKWOnDSbJkrzr7o -0gHv1bqwz5Q+gtAB1ktxgAMKEkWGeYHYS9LotbcsbJpMihAQ5OT1386EkcrRRuCE -cxdm22lDeICDvhRN8T4a1mz13kXzOa+R1T1URL9BzHNoGvwk25ECAwEAAaNQME4w -HQYDVR0OBBYEFJrPz27rcT3bPPGuiGtWcgPLCKdIMB8GA1UdIwQYMBaAFJrPz27r -cT3bPPGuiGtWcgPLCKdIMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB -ABAlyNwMVVzLg25573fsDY4MBsFLDNb3dVIhuBdKOIids3jEQvu4fBQ4EPus2hEA -W0KHXkWfbU5CpJoYBjkPRaaWiTLWWbPTjuOVtsSiS3QvZ8H7u/lybzdK5/RIM3Hf -uPXmQT/V1S8mCfgOkv9w6varWPuQBNZDLo+x+waradDcqPhbB/LUZh9j+F3BnkFE -u8nofeBG5KfIMl8xYuUcXIndt6JPng0TuF+xhFNMH84Z4QEAXr9BVZSppRPb8lnz -1k65nZ25CtmyGG18sfeWqr32+ZUPSm48fEZb39R47Jrc4uMB5oh3OZOcuipj+SVL -T6wIeTnGe98HNbrAwlC/WoE= +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA4MjYx +NDIzMTZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg +Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCAaIwDQYJKoZI +hvcNAQEBBQADggGPADCCAYoCggGBAJftVUG6NheV23Ec0+FhrFhz48aWzysfuAj1 +nUtLxzD2uAuzUnKgu8lNO47fIo4BV4HJknPMAMbscLA6F0DB3/KMNkzEp4HntiRo +4qB+NQcvoFv5RUb3HvBGEf7KGjxQ8SapX5winPhB4d9PEpUZL1yQARdufj59z+kJ +ryX4+EJ3LW1fNvJ4Hn1Kh2hjbAZxG436Jf7U0/WlF7Hv6hfLVMgnmYDLPEXxLFIc +3R9RRSBQHl6rV3MbQXiW3oSket2PMIU2WHl2oNJhyBuplJljxu74FL+0UlYxl/rr +rFOelc5MxFpKt8oDJ1s1V84C3OzKafiKWjnLFiAVAyRhbPR6/LZI5VkQXEnQI5/7 +cV466WifNHKAJ7Y/TLHZ22N/Z2hKbhH46MD0WhY5Uwto3nf6Ref4kc14zSiUl3FU ++8/wN97JJsXcG56JbQmsyERxy23xlzHVTCAzv3VKoODcaRHtKrRkEBEwiw6wpxDY +isWqG8gmiiXnZp+lahovfF+DxnhPHwIDAQABo1AwTjAdBgNVHQ4EFgQU3b/K2ubR +NLo3dSHKb5oIKPI1tkgwHwYDVR0jBBgwFoAU3b/K2ubRNLo3dSHKb5oIKPI1tkgw +DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAYEAM2pU02vA1wFfnfQFwZNm +kFDQtxjpsB5KoLbadpOvhNutFVQxFRPk3n5ODNUJHDQ1tuVM1m9lfTJf6/ypawf3 +SYLlgX4HgJpj+CzDQLyP1CraPtHuCLdNp4TK9D+hmEW+sQVp59/Xmasb7oswzPf8 +59QLF66Xv+R7/Q+ntIV541niFoe/HylFLCOTdr7Ahx3e7CtCauW7yPQKSggKjFzY +fU3RuL/V9yntktGUBOg1Bld/LCOXh6U1jSbTGkfyFtfZxtQfI0PTJpk5yiD0cSNv +DEp2dvd2H7P+v0ew/CpWgeHS3e4I2PT/WtwlYYqRArmGHPJQc3YlNfy2JSYVy+vE +K2EMHOfuLxeb7PDUoYTn0q/e5BskFKcBh+OrKVhGoNnACuCN11nTG/hUID54paXI +T4sDxJaf7PtHz3YtjWU0J7/6rgEFivOSCt2JbJehx+dgUees60t9LLhlyf5dakhV +juTH+WpA4bhkRem1WSmlX899WH1keeWkCawedmU9lMRo -----END CERTIFICATE----- diff --git a/Lib/test/idnsans.pem b/Lib/test/idnsans.pem index b4a771ce7ec1..e8a41fe5e565 100644 --- a/Lib/test/idnsans.pem +++ b/Lib/test/idnsans.pem @@ -1,64 +1,84 @@ -----BEGIN PRIVATE KEY----- -MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDeU8YQtyeEjPLA -SdMBPTS9QcuAZIJjbJRgr8nsRb767pbmWR9C1JuDy/Bz/AprFC6Om950fLn3pOqR -zDUWZX/qTe+o27i8u0Qzk06bhRkxAdTEoTfRcH/FkJaimJqeTt9rZqc+AGSNKM8o -4GyPW4IELnavmMB30+7rKIJMIpIn1a1k6MybJYdWNSuVqwArAVvRlj5qOiqX7KAS -otFRP8pz+Lgw3qREQzgnZz/bcScKd+5Uy4qMFPNOMjgW6nDV60ekNx0GT+59E/+8 -64GRq34rNVu2SN0XXcQh33R3LwwrvAdymaLyr1YyIRM5gLPxugxCIA0SYjG0YoGB -uUSwtNa7AgMBAAECggEBAJjxUGPXW1wYCja1km5byJgZVwEwI3J6E2igBWyAXm0J -DM3RqWu0DneQKA3h6NjYvV5lY5cG5nex/5vkuvB5SpHIo4GqBV/wA27ne0AJQ9cu -x0utDFUL6xnh6X5ZNKSK5a9gotRIOOPSmxAnswa7kKmHvSX3ExBbvxQOffQaJCk5 -0GHl6I/HltqVzMu4ICAo0NY0gw1n+hVKTo28KkJ9PL7X6v6H5yvZ3L6TkMytSvqf -9iVlYuIN66ToBtxaI4g2RiUJtA2hdT9IP7Wg4YD6Ptyih90zXz2wTzWppFem6UA9 -dePig94R9moj9ucuK0tx3kSATNo0op/XEx1e3OOtcQECgYEA/w7pNOPYgj7VMyYx -p4Lx4BOllzQts8mIBtUVZVQSJ2miun6DTalZVT2V3ayTuE0qhUHd1SHu9F77a9fQ -qaSUUY9elwXyfvcNCfhYVRJxyxirI4Z6ZCBwjpWOGSBB59NTeDhVnbkTlfE6guqS -3KRS1pfIQ6FCvGIrhjRZgHo1TGECgYEA3yXsospbOS7VeBj0UPSB87fp1QM+r48o -RflIsRzdsN9Ka2j6EiYpgKdbgXr80vkctYTK0dT8jrFSk81Y932CZezH2IWo8Meo -40qaFWMboNFBIC4yv6RSRxJMQfYsKnXC2trSnXH+qf55Trey4uZNMX7VJ+RFKExS -ieSWSbTWmJsCgYEAzo3yyoRiiEf+PKgHulLPMtp2VddJ07m30WCrLR5CfWyM/l8K -UtB8qg1v2s+x6aWEc9p9necXLwvkrNdgAqJoAw0KW1/TnILSKmrWjj6brRBTODfl -0kR7It128F4xQV7g0BE/NLX3aIytB+yT9t+Uvni5FBv6gbk26j5m5ScTFsECgYEA -hzrQYQcIqWq8av+Ub8r9Rdlal4BT6Mh0u5MKfmrj3mAzFUyU35LI6/J//cOum5vj -zg0fbHIKa98CEBgNpk4lS+dmZMz7SI92xedb4UIiaB7nvLzCfGj0g6WPGRo6QbED -2OVrZYbDsflJQm8ItYCjny8htf8b+gPmsTIZ8ajps6kCgYBnES8waDDAkL98lK28 -dcgnJXN+1UzeI6//If2uvDZEQ9tG/yMk2JYc84qZJLU5bRplMAjIQUVUcFWa+ZzV -ylnDhagAtiWkHPcElWHym9dH8CRuYM3OTDsApZ7yMB/ArCcZMIA35OvNf6uc4lNV -VD9VkaygPIg6ilv4npeTceqp8A== +MIIG/wIBADANBgkqhkiG9w0BAQEFAASCBukwggblAgEAAoIBgQDAfkvfgnT/+ATq +IiR5ksgGW4EJK900MkBsY9TXQcNdbrGqbyG4EwEvbDWzsAWBqgg76Alto7JgxKl2 +gNskz3T007xycOVAy4z31gzEWMTx+GxNbCiicJfCZjw6Z4Ck/UipooXAAy/S7jwC +z369E4RGa5jsIxxJ+NdDqAheoROIeUKawkfF7xXTyzkwuTuNj2YcV/TvJcsGVGQT +j4tZVSbDVvL38PlQX62VoGmEDK8pl8GL2GUQK3y+gP2VFDHepGxqqXxPLITxpdBF +aIoQcY4YV+nJtNu8vlUzDMizCHKII+YP9JdstxOLCyEbmjLv3rk4tHgIJuOfOyRh +3KjIcdghmNZcZ7vOvpbr341gKBNRLVmsAxtJCqqMfWvNOc6rXfp20bCnL/SDDJeB +/71/KfVeVaffXRtfEWcwJW6YRp7p2MealzAsnUA5xscOKItYhvmJ1FJlUB8LDZ+d +WESld2a4CIdK9lpuf3LDqy+wmcGTdnoVoi2RT68SDYHAb/bbtx0CAwEAAQKCAYEA +qNuhjzYPkKNvHDxLJuKd2QXYEcWKzEjwvqRsh0/HI2UBMXM/bwws0/7mtPPKQM55 +XqPT8XvR9JVP2wZ6NrZN8J71BqcGVGq7BIWeuXbtfR3hU6F3m9eNKRkp6En3TcRq +Q3SwQ5oID86wkf6XPSGrvWKRyUObg0BpBY6XmPtP+T32MXAQKg/rY9k4GAfFuA59 +EowN7uOGT80j5Qzcvsa1tFHH3bxj2B+2VDVdmlrOJXxcamKvdRnQ2gSpbmdCv/4X +HKmRiP7m2qVMxRaNh3kcL/gACGvZw3rUE7K0PwjW+sQ6lRlWF9IqUSDZaswxWrie +EaM2et9qD80g3eUV3gMi6I8BIdmD20p/yqa/l0bQk75Y6KQRUlN5HTodFL1u4/7V +W9N9wVIlx5DB1yJBkCZBDoebr4Y1UOGs4rrkcOd6F+zCeRSwKPFAng5vHpZ8jclS +1X2pV/KnKWIUfcFH9+v6pD9Y+VC5tz5LgaFz4SYFxgMF69t+PCpdoR2+0zwSWKGB +AoHBAPr/bC0L7V/HXeoiNxXjOSAYDgyVvzZC3i8cbW2/7DqdY7DfDmZbabdEX1mu +kaY9S/5abVV5W1eQACh1Cd/MaRkUbm+5nTy/bF9jXfh8408MzrGbAR9KF9GfWgqC +UxRiAkCa2gLuAhfc6uqokf5yZK0F+kZGGYWEeGcFK46i+8JstqknW0LsN1hhd+wE +PFokf+/22NiV0t1T0GERz18fFu0XoQC60D9qaW56EI+6OTADw71XkVh35jbNBa96 +bLPVzQKBwQDEVGMtpARdwoU59fRfDBo9S8iJJ/jTFpJs5nrqGZKah4J2pkTBiDpW +ZNxesVKL6IuvWghIuZGnBw5ovWT9JKQ2NHD4GDm/XJ6veVWOxc7THiBhCvJaG5N7 +3Jbxyn+eJ5yBO9PeR6slZbDrgDSIFUI++XD9A7abod8eTZHh6Q3pYT5mRuvfuDS1 +nlrkvMgGC1ZCQXCVTdq8MmMS9jea4UXhJLn4paGSKQSYXWLWVxTX3Mgk0640XpXN +HvsdCBMgFpECgcEAtESQkAXZ9Yk9tTPftlOcqqU8KeO/EhyScOmM2l8WLb7uY5DA +SdlrotlzVfjf2LJ7ivGtRs9DQC9HPcdZUSgBb6bR7mI0QPYGwreePSKNKzA6nyqB +ctSHKOG3DIcvXhtKHNZar/H7BOKAwgpR2JqWHgKGDsS3/cH3q531+5qpEKl9wx1m +GTmIQmy2cYdVRTSjwlfaxCtO5/ETXzQoaNvuXlgzJVAG0xT9JgB7No78FbUOd0WW +OSDb8g3sAGz/92UdAoHBAJqaOV9/gVC0FWWHIWCXjzIls00OhQr1A2luNeyTJrc5 +bqswU4RzWj0eP8rxqCc0gIsdQtjCCMiW7enc+uG9MPLeaPd1XEQtE2jrC0zax/vI +BYdlr2L1xDYNh2/wHn7UgpHiIv0vQr64TZGl67gdxYUBuNAFVI9J8QFT5EUZ2TKm +uRShVK1fyWIbyUfGwVzBILQcBCQOoukAZLRR34e/Td/1o90B0THfyxaQs3xGJsu1 +uIYHi28B7MUSWHyMF0N1gQKBwB9jtS7k7ZRgb0WzsgutRzJLfirmMeahzJwAQ46o +2xRT1y7vr0Idh9OSTz8Iqva6Frse21HT6agymamqEeC2aYwq+ikA0V7K3AhG3PiN +S5diC+RIZybeTthBTO1yj5pKDniZl+NGe3vIunXMrjZIuk/03VFK2etqdYcXguwH +SkwvuKqC3vvzcJW8dV2qIJ3Bz9LWz0QKqJQZBlCZvfzHf1jN5rpJJPQezys4k723 +ZPGg6Smyks3ElGXevVdidVcPUA== -----END PRIVATE KEY----- Certificate: Data: Version: 3 (0x2) Serial Number: - 82:ed:bf:41:c8:80:91:9f - Signature Algorithm: sha1WithRSAEncryption + cb:2d:80:99:5a:69:52:5f + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 19 19:09:07 2018 GMT - Not After : Nov 28 19:09:07 2027 GMT + Not Before: Aug 29 14:23:17 2018 GMT + Not After : Jul 7 14:23:17 2028 GMT Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=idnsans Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (2048 bit) + Public-Key: (3072 bit) Modulus: - 00:de:53:c6:10:b7:27:84:8c:f2:c0:49:d3:01:3d: - 34:bd:41:cb:80:64:82:63:6c:94:60:af:c9:ec:45: - be:fa:ee:96:e6:59:1f:42:d4:9b:83:cb:f0:73:fc: - 0a:6b:14:2e:8e:9b:de:74:7c:b9:f7:a4:ea:91:cc: - 35:16:65:7f:ea:4d:ef:a8:db:b8:bc:bb:44:33:93: - 4e:9b:85:19:31:01:d4:c4:a1:37:d1:70:7f:c5:90: - 96:a2:98:9a:9e:4e:df:6b:66:a7:3e:00:64:8d:28: - cf:28:e0:6c:8f:5b:82:04:2e:76:af:98:c0:77:d3: - ee:eb:28:82:4c:22:92:27:d5:ad:64:e8:cc:9b:25: - 87:56:35:2b:95:ab:00:2b:01:5b:d1:96:3e:6a:3a: - 2a:97:ec:a0:12:a2:d1:51:3f:ca:73:f8:b8:30:de: - a4:44:43:38:27:67:3f:db:71:27:0a:77:ee:54:cb: - 8a:8c:14:f3:4e:32:38:16:ea:70:d5:eb:47:a4:37: - 1d:06:4f:ee:7d:13:ff:bc:eb:81:91:ab:7e:2b:35: - 5b:b6:48:dd:17:5d:c4:21:df:74:77:2f:0c:2b:bc: - 07:72:99:a2:f2:af:56:32:21:13:39:80:b3:f1:ba: - 0c:42:20:0d:12:62:31:b4:62:81:81:b9:44:b0:b4: - d6:bb + 00:c0:7e:4b:df:82:74:ff:f8:04:ea:22:24:79:92: + c8:06:5b:81:09:2b:dd:34:32:40:6c:63:d4:d7:41: + c3:5d:6e:b1:aa:6f:21:b8:13:01:2f:6c:35:b3:b0: + 05:81:aa:08:3b:e8:09:6d:a3:b2:60:c4:a9:76:80: + db:24:cf:74:f4:d3:bc:72:70:e5:40:cb:8c:f7:d6: + 0c:c4:58:c4:f1:f8:6c:4d:6c:28:a2:70:97:c2:66: + 3c:3a:67:80:a4:fd:48:a9:a2:85:c0:03:2f:d2:ee: + 3c:02:cf:7e:bd:13:84:46:6b:98:ec:23:1c:49:f8: + d7:43:a8:08:5e:a1:13:88:79:42:9a:c2:47:c5:ef: + 15:d3:cb:39:30:b9:3b:8d:8f:66:1c:57:f4:ef:25: + cb:06:54:64:13:8f:8b:59:55:26:c3:56:f2:f7:f0: + f9:50:5f:ad:95:a0:69:84:0c:af:29:97:c1:8b:d8: + 65:10:2b:7c:be:80:fd:95:14:31:de:a4:6c:6a:a9: + 7c:4f:2c:84:f1:a5:d0:45:68:8a:10:71:8e:18:57: + e9:c9:b4:db:bc:be:55:33:0c:c8:b3:08:72:88:23: + e6:0f:f4:97:6c:b7:13:8b:0b:21:1b:9a:32:ef:de: + b9:38:b4:78:08:26:e3:9f:3b:24:61:dc:a8:c8:71: + d8:21:98:d6:5c:67:bb:ce:be:96:eb:df:8d:60:28: + 13:51:2d:59:ac:03:1b:49:0a:aa:8c:7d:6b:cd:39: + ce:ab:5d:fa:76:d1:b0:a7:2f:f4:83:0c:97:81:ff: + bd:7f:29:f5:5e:55:a7:df:5d:1b:5f:11:67:30:25: + 6e:98:46:9e:e9:d8:c7:9a:97:30:2c:9d:40:39:c6: + c7:0e:28:8b:58:86:f9:89:d4:52:65:50:1f:0b:0d: + 9f:9d:58:44:a5:77:66:b8:08:87:4a:f6:5a:6e:7f: + 72:c3:ab:2f:b0:99:c1:93:76:7a:15:a2:2d:91:4f: + af:12:0d:81:c0:6f:f6:db:b7:1d Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: @@ -70,11 +90,11 @@ Certificate: X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: - 3B:F0:22:A0:1E:9B:CE:2A:7C:AE:B1:32:1B:B0:8E:3E:33:40:E3:FA + 54:53:0C:3C:4C:E3:63:C6:56:08:35:9C:5E:F4:C1:A1:04:3A:C2:C9 X509v3 Authority Key Identifier: - keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48 + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server - serial:82:ED:BF:41:C8:80:91:9B + serial:CB:2D:80:99:5A:69:52:5B Authority Information Access: CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer @@ -85,52 +105,65 @@ Certificate: Full Name: URI:http://testca.pythontest.net/testca/revocation.crl - Signature Algorithm: sha1WithRSAEncryption - 8b:1f:d7:e4:0d:15:76:b4:f5:87:33:de:b9:84:9b:f2:c1:9b: - c9:97:50:f7:18:33:ed:b7:60:83:be:bb:94:1c:49:39:ae:54: - 24:43:f7:85:d8:2a:8c:26:17:56:1e:a6:b7:63:c5:05:f1:6e: - f4:79:eb:fd:af:12:84:3c:28:4a:8f:b1:01:97:91:ba:18:2b: - ba:54:25:49:1b:5b:2e:1e:6b:33:2d:f5:07:2e:76:04:e0:a8: - 95:25:3f:cc:c8:26:c0:30:b6:90:d2:2b:e1:e2:13:b0:a8:76: - f0:06:90:b9:d5:28:6b:8a:e9:72:1a:ed:4f:7e:3c:37:2e:00: - aa:9b:f1:29:44:94:f2:dc:c8:31:5f:4c:2d:00:d3:5e:78:6c: - 68:fc:0e:1e:46:be:d8:2e:29:88:78:8e:7e:f5:50:c8:5c:5d: - 5f:4c:09:d5:51:07:40:be:9b:30:ed:a3:29:68:25:6b:88:69: - c7:43:35:54:2f:6e:9a:30:f1:d6:87:54:84:20:ef:a5:aa:33: - df:00:6a:87:a9:b4:d7:89:1f:e7:60:0d:01:60:66:11:61:3f: - d0:9f:86:37:cc:b3:b8:48:7e:1f:d2:7a:0f:02:e7:11:1d:dd: - 34:c4:0b:45:47:2b:05:37:dd:ee:6e:0e:1c:bd:de:24:42:50: - a4:07:af:e5 + Signature Algorithm: sha256WithRSAEncryption + 03:af:93:ab:58:98:74:3c:a1:68:53:18:13:be:56:60:77:d0: + a7:0e:c9:8a:02:1f:92:4d:21:18:61:d7:3e:9a:1d:aa:94:37: + 02:59:31:3b:71:62:d9:cb:04:e8:c7:44:41:f9:0e:0d:90:d1: + 4a:ab:4e:e2:bd:4e:60:1b:3c:eb:2a:b7:8a:ca:24:58:60:64: + d3:a6:20:f6:f5:98:6e:ca:5a:0a:91:63:97:58:f5:18:90:9f: + 17:55:9f:12:cf:22:a7:31:e6:90:da:36:61:3b:d6:42:e6:18: + 19:43:bb:17:52:28:40:a5:11:47:7f:32:fe:83:3d:c3:7d:8b: + 17:1b:5d:f7:20:3e:bd:3b:16:3d:00:01:68:ed:76:7a:a1:af: + ce:cd:dd:52:7f:19:22:db:83:c1:89:b6:90:02:4e:22:bc:b1: + 76:0c:00:ad:c5:3c:33:be:64:92:30:38:1f:b1:04:4f:53:a2: + 4a:fb:63:80:21:8b:03:72:72:4b:df:f1:9d:08:75:f0:94:06: + d3:cb:7c:df:3b:80:19:58:7a:85:ca:2a:bf:b4:8c:5d:f3:b6: + 65:24:37:b2:1d:46:da:1f:39:48:f2:eb:3a:84:98:3a:0c:59: + ff:0a:05:c2:2c:8c:b1:a5:a9:43:a9:8e:47:97:9a:d2:40:9c: + b8:c3:e6:46:1b:db:4b:85:a3:e1:e7:4e:2a:1e:b9:a6:de:ee: + a7:f0:63:3f:0f:e2:90:b6:82:70:4b:93:7f:e9:74:f3:ab:03: + 9e:04:38:f1:46:2d:f6:fe:77:0f:4a:8e:66:23:74:3e:c6:5b: + eb:0e:dd:72:c3:46:1d:a4:f0:2f:b6:18:2e:f1:1c:f9:85:69: + 0e:5a:37:5e:f8:ea:4b:2d:e7:63:ee:a1:e4:b9:f7:fa:a9:11: + 65:64:a1:b7:b5:91:81:5c:4d:b5:27:b4:16:96:4b:df:fc:c4: + 8a:26:b6:87:62:54:88:fb:1d:7d:af:97:25:65:27:38:1e:f7: + 8c:a4:16:46:f2:d9 -----BEGIN CERTIFICATE----- -MIIFvTCCBKWgAwIBAgIJAILtv0HIgJGfMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIGvTCCBSWgAwIBAgIJAMstgJlaaVJfMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDdaFw0yNzExMjgx -OTA5MDdaMF0xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTdaFw0yODA3MDcx +NDIzMTdaMF0xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xEDAOBgNVBAMMB2lk -bnNhbnMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDeU8YQtyeEjPLA -SdMBPTS9QcuAZIJjbJRgr8nsRb767pbmWR9C1JuDy/Bz/AprFC6Om950fLn3pOqR -zDUWZX/qTe+o27i8u0Qzk06bhRkxAdTEoTfRcH/FkJaimJqeTt9rZqc+AGSNKM8o -4GyPW4IELnavmMB30+7rKIJMIpIn1a1k6MybJYdWNSuVqwArAVvRlj5qOiqX7KAS -otFRP8pz+Lgw3qREQzgnZz/bcScKd+5Uy4qMFPNOMjgW6nDV60ekNx0GT+59E/+8 -64GRq34rNVu2SN0XXcQh33R3LwwrvAdymaLyr1YyIRM5gLPxugxCIA0SYjG0YoGB -uUSwtNa7AgMBAAGjggKOMIICijCB4QYDVR0RBIHZMIHWggdpZG5zYW5zgh94bi0t -a25pZy01cWEuaWRuLnB5dGhvbnRlc3QubmV0gi54bi0ta25pZ3Nnc3NjaGVuLWxj -YjB3LmlkbmEyMDAzLnB5dGhvbnRlc3QubmV0gi54bi0ta25pZ3NnY2hlbi1iNGEz -ZHVuLmlkbmEyMDA4LnB5dGhvbnRlc3QubmV0giR4bi0tbnhhc21xNmIuaWRuYTIw -MDMucHl0aG9udGVzdC5uZXSCJHhuLS1ueGFzbW0xYy5pZG5hMjAwOC5weXRob250 -ZXN0Lm5ldDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG -AQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFDvwIqAem84qfK6xMhuwjj4z -QOP6MH0GA1UdIwR2MHSAFJrPz27rcT3bPPGuiGtWcgPLCKdIoVGkTzBNMQswCQYD -VQQGEwJYWTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0Ex -FjAUBgNVBAMMDW91ci1jYS1zZXJ2ZXKCCQCC7b9ByICRmzCBgwYIKwYBBQUHAQEE -dzB1MDwGCCsGAQUFBzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rl -c3RjYS9weWNhY2VydC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2EucHl0 -aG9udGVzdC5uZXQvdGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6 -Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3JsMA0G -CSqGSIb3DQEBBQUAA4IBAQCLH9fkDRV2tPWHM965hJvywZvJl1D3GDPtt2CDvruU -HEk5rlQkQ/eF2CqMJhdWHqa3Y8UF8W70eev9rxKEPChKj7EBl5G6GCu6VCVJG1su -HmszLfUHLnYE4KiVJT/MyCbAMLaQ0ivh4hOwqHbwBpC51ShriulyGu1Pfjw3LgCq -m/EpRJTy3MgxX0wtANNeeGxo/A4eRr7YLimIeI5+9VDIXF1fTAnVUQdAvpsw7aMp -aCVriGnHQzVUL26aMPHWh1SEIO+lqjPfAGqHqbTXiR/nYA0BYGYRYT/Qn4Y3zLO4 -SH4f0noPAucRHd00xAtFRysFN93ubg4cvd4kQlCkB6/l +bnNhbnMwggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDAfkvfgnT/+ATq +IiR5ksgGW4EJK900MkBsY9TXQcNdbrGqbyG4EwEvbDWzsAWBqgg76Alto7JgxKl2 +gNskz3T007xycOVAy4z31gzEWMTx+GxNbCiicJfCZjw6Z4Ck/UipooXAAy/S7jwC +z369E4RGa5jsIxxJ+NdDqAheoROIeUKawkfF7xXTyzkwuTuNj2YcV/TvJcsGVGQT +j4tZVSbDVvL38PlQX62VoGmEDK8pl8GL2GUQK3y+gP2VFDHepGxqqXxPLITxpdBF +aIoQcY4YV+nJtNu8vlUzDMizCHKII+YP9JdstxOLCyEbmjLv3rk4tHgIJuOfOyRh +3KjIcdghmNZcZ7vOvpbr341gKBNRLVmsAxtJCqqMfWvNOc6rXfp20bCnL/SDDJeB +/71/KfVeVaffXRtfEWcwJW6YRp7p2MealzAsnUA5xscOKItYhvmJ1FJlUB8LDZ+d +WESld2a4CIdK9lpuf3LDqy+wmcGTdnoVoi2RT68SDYHAb/bbtx0CAwEAAaOCAo4w +ggKKMIHhBgNVHREEgdkwgdaCB2lkbnNhbnOCH3huLS1rbmlnLTVxYS5pZG4ucHl0 +aG9udGVzdC5uZXSCLnhuLS1rbmlnc2dzc2NoZW4tbGNiMHcuaWRuYTIwMDMucHl0 +aG9udGVzdC5uZXSCLnhuLS1rbmlnc2djaGVuLWI0YTNkdW4uaWRuYTIwMDgucHl0 +aG9udGVzdC5uZXSCJHhuLS1ueGFzbXE2Yi5pZG5hMjAwMy5weXRob250ZXN0Lm5l +dIIkeG4tLW54YXNtbTFjLmlkbmEyMDA4LnB5dGhvbnRlc3QubmV0MA4GA1UdDwEB +/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/ +BAIwADAdBgNVHQ4EFgQUVFMMPEzjY8ZWCDWcXvTBoQQ6wskwfQYDVR0jBHYwdIAU +3b/K2ubRNLo3dSHKb5oIKPI1tkihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQK +DB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNh +LXNlcnZlcoIJAMstgJlaaVJbMIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKG +MGh0dHA6Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNl +cjA1BggrBgEFBQcwAYYpaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0 +Y2Evb2NzcC8wQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250 +ZXN0Lm5ldC90ZXN0Y2EvcmV2b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQELBQADggGB +AAOvk6tYmHQ8oWhTGBO+VmB30KcOyYoCH5JNIRhh1z6aHaqUNwJZMTtxYtnLBOjH +REH5Dg2Q0UqrTuK9TmAbPOsqt4rKJFhgZNOmIPb1mG7KWgqRY5dY9RiQnxdVnxLP +Iqcx5pDaNmE71kLmGBlDuxdSKEClEUd/Mv6DPcN9ixcbXfcgPr07Fj0AAWjtdnqh +r87N3VJ/GSLbg8GJtpACTiK8sXYMAK3FPDO+ZJIwOB+xBE9Tokr7Y4AhiwNyckvf +8Z0IdfCUBtPLfN87gBlYeoXKKr+0jF3ztmUkN7IdRtofOUjy6zqEmDoMWf8KBcIs +jLGlqUOpjkeXmtJAnLjD5kYb20uFo+HnTioeuabe7qfwYz8P4pC2gnBLk3/pdPOr +A54EOPFGLfb+dw9KjmYjdD7GW+sO3XLDRh2k8C+2GC7xHPmFaQ5aN1746kst52Pu +oeS59/qpEWVkobe1kYFcTbUntBaWS9/8xIomtodiVIj7HX2vlyVlJzge94ykFkby +2Q== -----END CERTIFICATE----- diff --git a/Lib/test/keycert.passwd.pem b/Lib/test/keycert.passwd.pem index 0ad696055190..cbb3c3bccd2c 100644 --- a/Lib/test/keycert.passwd.pem +++ b/Lib/test/keycert.passwd.pem @@ -1,50 +1,68 @@ -----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED -DEK-Info: DES-EDE3-CBC,E74528136B90D2DD +DEK-Info: DES-EDE3-CBC,D134E931C96D9DEC -WRHVD2PJXPqjFSHg92HURIsUzvsTE4a9oi0SC5yMBFKNWA5Z933gK3XTifp6jul5 -zpNYi8jBXZ2EqJJBxCuVcefmXSxL0q7CMej25TdIC4BVAFJVveeprHPUFkNB0IM1 -go5Lg4YofYqTCg3OE3k7WvfR3Zg1cRYxksDKO+WNZgWyKBex5X4vjOiyUqDl3GKt -kQXnkg1VgPV2Vrx93S9XNdELNRTguwf+XG0fkhtYhp/zCto8uKTgy5elK2P/ulGp -7fe6uj7h/uN9L7EOC6CjRkitywfeBUER739mOcGT4imSFJ9G27TCqPzj2ea3uuaf -/v1xhkQ4M6lNY/gcRfgVpCXhW43aAQV8XXQRMJTqLmz5Y5hYTKn+Ugq5vJ/ngyRM -lu1gUJnYYaemBTb4hbm6aBvnYK9mORa891Pmf+vxU9rYuQIdVAhvvXh4KBreSEBI -1AFy6dFKXl8ZKs6Wrq5wPefmFFkRmZ8OBiiq0fp2ApCRGZw6LsjCgwrRM38JiY7d -3OdsJpKvRYufgUyuuzUE0xA+E4yMvD48M9pPq2fC8O5giuGL1uEekQWXJuq+6ZRI -XYKIeSkuQALbX3RAzCPXTUEMtCYXKm/gxrrwJ+Bet4ob2amf3MX0uvWwOuAq++Fk -J0HFSBxrwvIWOhyQXOPuJdAN8PXA7dWOXfOgOMF0hQYqZCl3T4TiVZJbwVQtg1sN -dO7oAD5ZPHiKzvveZuB6k1FlBG8j0TyAC+44ChxkPDD3jF4dd6zGe62sDf85p4/d -W80gxJeD3xnDxG0ePPns+GuKUpUaWS7WvHtDpeFW1JEhvOqf8p1Li9a7RzWVo8ML -mGTdQgBIYIf6/fk69pFKl0nKtBU75KaunZz4nAmd9bNED4naDurMBg44u5TvODbJ -vgYIYXIYjNvONbskJatVrrTS8zch2NwVIjCi8L/hecwBXbIXzo1pECpc6BU7sQT8 -+i9sDKBeJcRipzfKZNHvnO19mUZaPCY8+a/f9c21DgKXz+bgLcJbohpSaeGM8Gfc -aZd3Vp9n3OJ3g2zQR1++HO9v1vR/wLELu6MeydkvMduHLmOPCn54gZ9z51ZNPAwa -qfFIsH+mLh9ks0H74ssF59uIlstkgB9zmZHv/Q0dK9ZfG/VEH6rSgdETWhZxhoMQ -Z92jXBEFT0zhI3rrIPNY+XS7eJCQIc1wc84Ea3cRk7SP+S1og3JtAxX56ykUwtkM -LQ/Dwwa6h1aqD0l2d5x1/BSdavtTuSegISRWQ4iOmSvEdlFP7H4g6RZk/okbLzMD -Evq5gNc7vlXhVawoQU8JCanJ5ZbbWnIRZfiXxBQS4lpYPKvJt4ML9z/x+82XxcXv -Z93N2Wep7wWW5OwS2LcQcOgZRDSIPompwo/0pMFGOS+5oort0ZDRHdmmGLjvBcCb -1KQmKQ4+8brI/3rjRzts6uDLjTGNxSCieNsnqhwHUv9Mg9WDSWupcGa+x27L89x3 -rObf6+3umcFLSjIzU8wuv1hx/e/y98Kv7BDBNYpAr6kVMrLnzYjAfJbBmqlxkzkQ -IgQzgrk2QZoTdgwR+S374NAMO0AE5IlO+/qa6qp2SORGTDX64I3UNw== +nuGFEej7vIjkYWSMz5OJeVTNntDRQi6ZM4DBm3g8T7i/0odr3WFqGMMKZcIhLYQf +rgRq7RSKtrJ1y5taVucMV+EuCjyfzDo0TsYt+ZrXv/D08eZhjRmkhoHnGVF0TqQm +nQEXM/ERT4J2RM78dnG+homMkI76qOqxgGbRqQqJo6AiVRcAZ45y8s96bru2TAB8 ++pWjO/v0Je7AFVdwSU52N8OOY6uoSAygW+0UY1WVxbVGJF2XfRsNpPX+YQHYl6e+ +3xM5XBVCgr6kmdAyub5qUJ38X3TpdVGoR0i+CVS9GTr2pSRib1zURAeeHnlqiUZM +4m0Gn9s72nJevU1wxED8pwOhR8fnHEmMKGD2HPhKoOCbzDhwwBZO27TNa1uWeM3f +M5oixKDi2PqMn3y2cDx1NjJtP661688EcJ5a2Ih9BgO9xpnhSyzBWEKcAn0tJB0H +/56M0FW6cdOOIzMveGGL7sHW5E+iOdI1n5e7C6KJUzew78Y9qJnhS53EdI6qTz9R +wsIsj1i070Fk6RbPo6zpLlF6w7Zj8GlZaZA7OZZv9wo5VEV/0ST8gmiiBOBc4C6Y +u9hyLIIu4dFEBKyQHRvBnQSLNpKx6or1OGFDVBay2In9Yh2BHh1+vOj/OIz/wq48 +EHOIV27fRJxLu4jeK5LIGDhuPnMJ8AJYQ0bQOUP6fd7p+TxWkAQZPB/Dx/cs3hxr +nFEdzx+eO+IAsObx/b1EGZyEJyETBslu4GwYX7/KK3HsJhDJ1bdZ//28jOCaoir6 +ZOMT72GRwmVoQTJ0XpccfjHfKJDRLT7C1xvzo4Eibth0hpTZkA75IUYUp6qK/PuJ +kH/qdiC7QIkRKtsrawW4vEDna3YtxIYhQqz9+KwO6u/0gzooZtv1RU4U3ifMDB5u +5P5GAzACRqlY8QYBkM869lvWqzQPHvybC4ak9Yx6/heMO9ddjdIW9BaK8BLxvN/6 +UCD936Y4fWltt09jHZIoxWFykouBwmd7bXooNYXmDRNmjTdVhKJuOEOQw8hDzx7e +pWFJ9Z/V4Qm1tvXbCD7QFqMCDoY3qFvVG8DBqXpmxe1yPfz21FWrT7IuqDXAD3ns +vxfN/2a+Cy04U9FBNVCvWqWIs5AgNpdCMJC2FlXKTy+H3/7rIjNyFyvbX0vxIXtK +liOVNXiyVM++KZXqktqMUDlsJENmIHV9B046luqbgW018fHkyEYlL3iRZGbYegwr +XO9VVIKVPw1BEvJ8VNdGFGuZGepd8qX2ezfYADrNR+4t85HDm8inbjTobSjWuljs +ftUNkOeCHqAvWCFQTLCfdykvV08EJfVY79y7yFPtfRV2gxYokXFifjo3su9sVQr1 +UiIS5ZAsIC1hBXWeXoBN7QVTkFi7Yto6E1q2k10LiT3obpUUUQ/oclhrJOCJVjrS +oRcj2QBy8OT4T9slJr5maTWdgd7Lt6+I6cGQXPaDvjGOJl0eBYM14vhx4rRQWytJ +k07hhHFO4+9CGCuHS8AAy2gR6acYFWt2ZiiNZ0z/iPIHNK4YEyy9aLf6uZH/KQjE +jmHToo7XD6QvCAEC5qTHby3o3LfHIhyZi/4L+AhS4FKUHF6M0peeyYt4z3HaK2d2 +N6mHLPdjwNjra7GOmcns4gzcrdfoF+R293KpPal4PjknvR3dZL4kKP/ougTAM5zv +qDIvRbkHzjP8ChTpoLcJsNVXykNcNkjcSi0GHtIpYjh6QX6P2uvR/S4+Bbb9p9rn +hIy/ovu9tWN2hiPxGPe6torF6BulAxsTYlDercC204AyzsrdA0pr6HBgJH9C6ML1 +TchwodbFJqn9rSv91i1liusAGoOvE81AGBdrXY7LxfSNhYY1IK6yR/POJPTd53sA +uX2/j6Rtoksd/2BHPM6AUnI/2B9slhuzWX2aCtWLeuwvXDS6rYuTigaQmLkzTRfM +dlMI3s9KLXxgi5YVumUZleJWXwBNP7KiKajd+VTSD+7WAhyhM5FIG5wVOaxmy4G2 +TyqZ/Ax9d2VEjTQHWvQlLPQ4Mp0EIz0aEl94K/S8CK8bJRH6+PRkar+dJi1xqlL+ +BYb42At9mEJ8odLlFikvNi1+t7jqXk5jRi5C0xFKx3nTtzoH2zNUeuA3R6vSocVK +45jnze9IkKmxMlJ4loR5sgszdpDCD3kXqjtCcbMTmcrGyzJek3HSOTpiEORoTFOe +Rhg6jH5lm+QcC263oipojS0qEQcnsWJP2CylNYMYHR9O/9NQxT3o2lsRHqZTMELV +uQa/SFH+paQNbZOj8MRwPSqqiIxJFuLswKte1R+W7LKn1yBSM7Pp39lNbzGvJD2E +YRfnCwFpJ54voVAuQ4jXJvigCW2qeCjXlxeD6K2j4eGJEEOmIjIW1wjubyBY6OI3 -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIIDWTCCAkGgAwIBAgIJAPm6B21bar2bMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV +MIIEWTCCAsGgAwIBAgIJAJinz4jHSjLtMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODAx -MTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH +IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODA4 +MjkxNDIzMTVaFw0yODA4MjYxNDIzMTVaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k -YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEP -ADCCAQoCggEBAKvvsX2gEti4shve3iYMc+jE4Se7WHs1Bol2f21H8qNboDOFdeb1 -RKHjmq3exHpajywOUEgne9nKHJY/3f2phR4Y5klqG6liLgiSpVyRlcBGbeT2qEAj -9oLiLFUXLGfGDds2mTwivQDLJBWi51j7ff5k2Pr58fN5ugYMn24T9FNyn0moT+qj -SFoBNm58l9jrdkJSlgWfqPlbiMa+mqDn/SFtrwLF2Trbfzu42Sd9UdIzMaSSrzbN -sGm53pNhCh8KndWUQ8GPP2IsLPoUU4qAtmZuTxCx2S1cXrN9EkmT69tlOH84YfSn -96Ih9bWRc7M5y5bfVdEVM+fKQl3hBRf05qMCAwEAAaMYMBYwFAYDVR0RBA0wC4IJ -bG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQAtQ8f37cCEk7/rAcbYR53ce3iK -Vpihb0U2ni1QjG9Tg9UIExkIGkwTiCm7kwQL+GEStBu9AG/QVrOjeTriRiddhWkk -ze8kRaI3AC/63t6Vh9Q1x6PESgeE4OtAO9JpJCf4GILglA789Y/b/GF8zJZQxR13 -qpB4ZwWw7gCBhdEW59u6CFeBmfDa58hM8lWvuVoRrTi7bjUeC6PAn5HVMzZSykhu -4HaUfBp6bKFjuym2+h/VvM1n8C3chjVSmutsLb6ELdD8IK0vPV/yf5+LN256zSsS -dyUZYd8XwQaioEMKdbhLvnehyzHiWfQIUR3BdhONxoIJhHv/EAo8eCkHHYIF +YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGP +ADCCAYoCggGBALKUqUtopT6E68kN+uJNEt34i2EbmG/bwjcD8IaMsgJPSsMO2Bpd +3S6qWgkCeOyCfmAwBxK2kNbxGb63ouysEv7l8GCTJTWv3hG/HQcejJpnAEGi6K1U +fDbyE/db6yZ12SoHVTGkadN4vYGCPd1Wj9ZO1F877SHQ8rDWX3xgTWkxN2ojBw44 +T8RHSDiG8D/CvG4uEy+VUszL+Uvny5y2poNSqvI3J56sptWSrh8nIIbkPZPBdUne +LYMOHTFK3ZjXSmhlXgziTxK71nnzM3Y9K9gxPnRqoXbvu/wFo55hQCkETiRkYgmm +jXcBMZ0TClQVnQWuLjMthRnWFZs4Lfmwqjs7FZD/61581R2BYehvpWbLvvuOJhwv +DFzexL2sXcAl7SsxbzeQKRHqGbIDfbnQTXfs3/VC6Ye5P82P2ucj+XC32N9piRmO +gCBP8L3ub+YzzdxikZN2gZXXE2jsb3QyE/R2LkWdWyshpKe+RsZP1SBRbHShUyOh +yJ90baoiEwj2mwIDAQABoxgwFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZI +hvcNAQELBQADggGBAHRUO/UIHl3jXQENewYayHxkIx8t7nu40iO2DXbicSijz5bo +5//xAB6RxhBAlsDBehgQP1uoZg+WJW+nHu3CIVOU3qZNZRaozxiCl2UFKcNqLOmx +R3NKpo1jYf4REQIeG8Yw9+hSWLRbshNteP6bKUUf+vanhg9+axyOEOH/iOQvgk/m +b8wA8wNa4ujWljPbTQnj7ry8RqhTM0GcAN5LSdSvcKcpzLcs3aYwh+Z8e30sQWna +F40sa5u7izgBTOrwpcDm/w5kC46vpRQ5fnbshVw6pne2by0mdMECASid/p25N103 +jMqTFlmO7kpf/jpCSmamp3/JSEE1BJKHwQ6Ql4nzRA2N1mnvWH7Zxcv043gkHeAu +0x8evpvwuhdIyproejNFlBpKmW8OX7yKTCPPMC/VkX8Q1rVkxU0DQ6hmvwZlhoKa +9Wc2uXpw9xF8itV4Uvcdr3dwqByvIqn7iI/gB+4l41e0u8OmH2MKOx4Nxlly5TNW +HcVKQHyOeyvnINuBAQ== -----END CERTIFICATE----- diff --git a/Lib/test/keycert.pem b/Lib/test/keycert.pem index 9545dcf4b94f..0d398633739a 100644 --- a/Lib/test/keycert.pem +++ b/Lib/test/keycert.pem @@ -1,48 +1,66 @@ -----BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCr77F9oBLYuLIb -3t4mDHPoxOEnu1h7NQaJdn9tR/KjW6AzhXXm9USh45qt3sR6Wo8sDlBIJ3vZyhyW -P939qYUeGOZJahupYi4IkqVckZXARm3k9qhAI/aC4ixVFyxnxg3bNpk8Ir0AyyQV -oudY+33+ZNj6+fHzeboGDJ9uE/RTcp9JqE/qo0haATZufJfY63ZCUpYFn6j5W4jG -vpqg5/0hba8Cxdk62387uNknfVHSMzGkkq82zbBpud6TYQofCp3VlEPBjz9iLCz6 -FFOKgLZmbk8QsdktXF6zfRJJk+vbZTh/OGH0p/eiIfW1kXOzOcuW31XRFTPnykJd -4QUX9OajAgMBAAECggEAHppmXDbuw9Z0FVPg9KLIysioTtsgz6VLiZIm8juZK4x2 -glUh/D7xvWL2uDXrgN+3lh7iGUW13LkFx5SMncbbo9TIwI57Z/XKvcnkVwquve+L -RfLFVc1Q5lD9lROv2rS86KTaN4LzYz3FKXi6dvMkpPAsUtfEQhMLkmISypQQq/1z -EJaqo7r85OjN7e0wKazlKZpOzJEa5FQLMVRjTRFhLFNbHXX/tAet2jw+umATKbw8 -hYgiuZ44TwSEd9JeIV/oSYWfI/3HetuYW0ru3caiztRF2NySNu8lcsWgNC7fIku9 -mcHjtSNzs91QN1Qlu7GQvvhpt6OWDirNDCW+49WGaQKBgQDg9SDhfF0jRYslgYbH -cqO4ggaFdHjrAAYpwnAgvanhFZL/zEqm5G1E7l/e2fCkJ9VOSFO0A208chvwMcr+ -dCjHE2tVdE81aQ2v/Eo83VdS1RcOV4Y75yPH48rMhxPaHvxWD/FFDbf0/P2mtPB7 -SZ3kIeZMkE1wxdaO3AKUbQoozwKBgQDDqYgg7kVtygyICE1mB8Hwp6nUxFTczG7y -4XcsDqMIrKmw+PbQluvkoHoStxeVrsTloDhkTjIrpmYLyAiazg+PUJdkd6xrfLSj -VV6X93W0S/1egEb1F1CGFxtk8v/PWH4K76EPL2vxXdxjywz3GWlrL9yDYaB2szzS -DqgwVMqx7QKBgDCD7UF0Bsoyl13RX3XoPXLvZ+SkR+e2q52Z94C4JskKVBeiwX7Y -yNAS8M4pBoMArDoj0xmBm69rlKbqtjLGbnzwrTdSzDpim7cWnBQgUFLm7gAD1Elb -AhZ8BCK0Bw4FnLoa2hfga4oEfdfUMgEE0W5/+SEOBgWKRUmuHUhRc911AoGAY2EN -YmSDYSM5wDIvVb5k9B3EtevOiqNPSw/XnsoEZtiEC/44JnQxdltIBY93bDBrk5IQ -cmoBM4h91kgQjshQwOMXMhFSwvmBKmCm/hrTbvMVytTutXfVD3ZXFKwT4DW7N0TF -ElhsxBh/YzRz7mG62JVjtFt2zDN3ld2Z8YpvtXUCgYEA4EJ4ObS5YyvcXAKHJFo6 -Fxmavyrf8LSm3MFA65uSnFvWukMVqqRMReQc5jvpxHKCis+XvnHzyOfL0gW9ZTi7 -tWGGbBi0TRJCa8BkvgngUZxOxUlMfg/7cVxOIB0TPoUSgxFd/+qVz4GZMvr0dPu7 -eAF7J/8ECVvb0wSPTUI1N3c= +MIIG/wIBADANBgkqhkiG9w0BAQEFAASCBukwggblAgEAAoIBgQCylKlLaKU+hOvJ +DfriTRLd+IthG5hv28I3A/CGjLICT0rDDtgaXd0uqloJAnjsgn5gMAcStpDW8Rm+ +t6LsrBL+5fBgkyU1r94Rvx0HHoyaZwBBouitVHw28hP3W+smddkqB1UxpGnTeL2B +gj3dVo/WTtRfO+0h0PKw1l98YE1pMTdqIwcOOE/ER0g4hvA/wrxuLhMvlVLMy/lL +58uctqaDUqryNyeerKbVkq4fJyCG5D2TwXVJ3i2DDh0xSt2Y10poZV4M4k8Su9Z5 +8zN2PSvYMT50aqF277v8BaOeYUApBE4kZGIJpo13ATGdEwpUFZ0Fri4zLYUZ1hWb +OC35sKo7OxWQ/+tefNUdgWHob6Vmy777jiYcLwxc3sS9rF3AJe0rMW83kCkR6hmy +A3250E137N/1QumHuT/Nj9rnI/lwt9jfaYkZjoAgT/C97m/mM83cYpGTdoGV1xNo +7G90MhP0di5FnVsrIaSnvkbGT9UgUWx0oVMjocifdG2qIhMI9psCAwEAAQKCAYBT +sHmaPmNaZj59jZCqp0YVQlpHWwBYQ5vD3pPE6oCttm0p9nXt/VkfenQRTthOtmT1 +POzDp00/feP7zeGLmqSYUjgRekPw4gdnN7Ip2PY5kdW77NWwDSzdLxuOS8Rq1MW9 +/Yu+ZPe3RBlDbT8C0IM+Atlh/BqIQ3zIxN4g0pzUlF0M33d6AYfYSzOcUhibOO7H +j84r+YXBNkIRgYKZYbutRXuZYaGuqejRpBj3voVu0d3Ntdb6lCWuClpB9HzfGN0c +RTv8g6UYO4sK3qyFn90ibIR/1GB9watvtoWVZqggiWeBzSWVWRsGEf9O+Cx4oJw1 +IphglhmhbgNksbj7bD24on/icldSOiVkoUemUOFmHWhCm4PnB1GmbD8YMfEdSbks +qDr1Ps1zg4mGOinVD/4cY7vuPFO/HCH07wfeaUGzRt4g0/yLr+XjVofOA3oowyxv +JAzr+niHA3lg5ecj4r7M68efwzN1OCyjMrVJw2RAzwvGxE+rm5NiT08SWlKQZnkC +gcEA4wvyLpIur/UB84nV3XVJ89UMNBLm++aTFzld047BLJtMaOhvNqx6Cl5c8VuW +l261KHjiVzpfNM3/A2LBQJcYkhX7avkqEXlj57cl+dCWAVwUzKmLJTPjfaTTZnYJ +xeN3dMYjJz2z2WtgvfvDoJLukVwIMmhTY8wtqqYyQBJ/l06pBsfw5TNvmVIOQHds +8ASOiFt+WRLk2bl9xrGGayqt3VV93KVRzF27cpjOgEcG74F3c0ZW9snERN7vIYwB +JfrlAoHBAMlahPwMP2TYylG8OzHe7EiehTekSO26LGh0Cq3wTGXYsK/q8hQCzL14 +kWW638vpwXL6L9ntvrd7hjzWRO3vX/VxnYEA6f0bpqHq1tZi6lzix5CTUN5McpDg +QnjenSJNrNjS1zEF8WeY9iLEuDI/M/iUW4y9R6s3WpgQhPDXpSvd2g3gMGRUYhxQ +Xna8auiJeYFq0oNaOxvJj+VeOfJ3ZMJttd+Y7gTOYZcbg3SdRb/kdxYki0RMD2hF +4ZvjJ6CTfwKBwQDiMqiZFTJGQwYqp4vWEmAW+I4r4xkUpWatoI2Fk5eI5T9+1PLX +uYXsho56NxEU1UrOg4Cb/p+TcBc8PErkGqR0BkpxDMOInTOXSrQe6lxIBoECVXc3 +HTbrmiay0a5y5GfCgxPKqIJhfcToAceoVjovv0y7S4yoxGZKuUEe7E8JY2iqRNAO +yOvKCCICv/hcN235E44RF+2/rDlOltagNej5tY6rIFkaDdgOF4bD7f9O5eEni1Bg +litfoesDtQP/3rECgcEAkQfvQ7D6tIPmbqsbJBfCr6fmoqZllT4FIJN84b50+OL0 +mTGsfjdqC4tdhx3sdu7/VPbaIqm5NmX10bowWgWSY7MbVME4yQPyqSwC5NbIonEC +d6N0mzoLR0kQ+Ai4u+2g82gicgAq2oj1uSNi3WZi48jQjHYFulCbo246o1NgeFFK +77WshYe2R1ioQfQDOU1URKCR0uTaMHClgfu112yiGd12JAD+aF3TM0kxDXz+sXI5 +SKy311DFxECZeXRLpcC3AoHBAJkNMJWTyPYbeVu+CTQkec8Uun233EkXa2kUNZc/ +5DuXDaK+A3DMgYRufTKSPpDHGaCZ1SYPInX1Uoe2dgVjWssRL2uitR4ENabDoAOA +ICVYXYYNagqQu5wwirF0QeaMXo1fjhuuHQh8GsMdXZvYEaAITZ9/NG5x/oY08+8H +kr78SMBOPy3XQn964uKG+e3JwpOG14GKABdAlrHKFXNWchu/6dgcYXB87mrC/GhO +zNwzC+QhFTZoOomFoqMgFWujng== -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIIDWTCCAkGgAwIBAgIJAPm6B21bar2bMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV +MIIEWTCCAsGgAwIBAgIJAJinz4jHSjLtMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODAx -MTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH +IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODA4 +MjkxNDIzMTVaFw0yODA4MjYxNDIzMTVaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k -YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEP -ADCCAQoCggEBAKvvsX2gEti4shve3iYMc+jE4Se7WHs1Bol2f21H8qNboDOFdeb1 -RKHjmq3exHpajywOUEgne9nKHJY/3f2phR4Y5klqG6liLgiSpVyRlcBGbeT2qEAj -9oLiLFUXLGfGDds2mTwivQDLJBWi51j7ff5k2Pr58fN5ugYMn24T9FNyn0moT+qj -SFoBNm58l9jrdkJSlgWfqPlbiMa+mqDn/SFtrwLF2Trbfzu42Sd9UdIzMaSSrzbN -sGm53pNhCh8KndWUQ8GPP2IsLPoUU4qAtmZuTxCx2S1cXrN9EkmT69tlOH84YfSn -96Ih9bWRc7M5y5bfVdEVM+fKQl3hBRf05qMCAwEAAaMYMBYwFAYDVR0RBA0wC4IJ -bG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQAtQ8f37cCEk7/rAcbYR53ce3iK -Vpihb0U2ni1QjG9Tg9UIExkIGkwTiCm7kwQL+GEStBu9AG/QVrOjeTriRiddhWkk -ze8kRaI3AC/63t6Vh9Q1x6PESgeE4OtAO9JpJCf4GILglA789Y/b/GF8zJZQxR13 -qpB4ZwWw7gCBhdEW59u6CFeBmfDa58hM8lWvuVoRrTi7bjUeC6PAn5HVMzZSykhu -4HaUfBp6bKFjuym2+h/VvM1n8C3chjVSmutsLb6ELdD8IK0vPV/yf5+LN256zSsS -dyUZYd8XwQaioEMKdbhLvnehyzHiWfQIUR3BdhONxoIJhHv/EAo8eCkHHYIF +YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGP +ADCCAYoCggGBALKUqUtopT6E68kN+uJNEt34i2EbmG/bwjcD8IaMsgJPSsMO2Bpd +3S6qWgkCeOyCfmAwBxK2kNbxGb63ouysEv7l8GCTJTWv3hG/HQcejJpnAEGi6K1U +fDbyE/db6yZ12SoHVTGkadN4vYGCPd1Wj9ZO1F877SHQ8rDWX3xgTWkxN2ojBw44 +T8RHSDiG8D/CvG4uEy+VUszL+Uvny5y2poNSqvI3J56sptWSrh8nIIbkPZPBdUne +LYMOHTFK3ZjXSmhlXgziTxK71nnzM3Y9K9gxPnRqoXbvu/wFo55hQCkETiRkYgmm +jXcBMZ0TClQVnQWuLjMthRnWFZs4Lfmwqjs7FZD/61581R2BYehvpWbLvvuOJhwv +DFzexL2sXcAl7SsxbzeQKRHqGbIDfbnQTXfs3/VC6Ye5P82P2ucj+XC32N9piRmO +gCBP8L3ub+YzzdxikZN2gZXXE2jsb3QyE/R2LkWdWyshpKe+RsZP1SBRbHShUyOh +yJ90baoiEwj2mwIDAQABoxgwFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZI +hvcNAQELBQADggGBAHRUO/UIHl3jXQENewYayHxkIx8t7nu40iO2DXbicSijz5bo +5//xAB6RxhBAlsDBehgQP1uoZg+WJW+nHu3CIVOU3qZNZRaozxiCl2UFKcNqLOmx +R3NKpo1jYf4REQIeG8Yw9+hSWLRbshNteP6bKUUf+vanhg9+axyOEOH/iOQvgk/m +b8wA8wNa4ujWljPbTQnj7ry8RqhTM0GcAN5LSdSvcKcpzLcs3aYwh+Z8e30sQWna +F40sa5u7izgBTOrwpcDm/w5kC46vpRQ5fnbshVw6pne2by0mdMECASid/p25N103 +jMqTFlmO7kpf/jpCSmamp3/JSEE1BJKHwQ6Ql4nzRA2N1mnvWH7Zxcv043gkHeAu +0x8evpvwuhdIyproejNFlBpKmW8OX7yKTCPPMC/VkX8Q1rVkxU0DQ6hmvwZlhoKa +9Wc2uXpw9xF8itV4Uvcdr3dwqByvIqn7iI/gB+4l41e0u8OmH2MKOx4Nxlly5TNW +HcVKQHyOeyvnINuBAQ== -----END CERTIFICATE----- diff --git a/Lib/test/keycert2.pem b/Lib/test/keycert2.pem index bb5fa65a8aca..ed6ae85a4649 100644 --- a/Lib/test/keycert2.pem +++ b/Lib/test/keycert2.pem @@ -1,49 +1,66 @@ -----BEGIN PRIVATE KEY----- -MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQC3ulRNfhbOAey/ -B+wIVYx+d5az7EV4riR6yi/qE6G+bxbTvay2pqySHtDweuaYSh2cVmcasBKKIFJm -rCD1zR8UmLb5i2XFIina1t3eePCuBZMrvZZwkzlQUSM1AZtjGOO/W0I3FwO6y645 -9xA5PduKI7SMYkH/VL3zE5W1JwMovv6bvNiT+GU5l6mB9ylCTgLpmUqoQhRqz/35 -zCzVyoh+ppDvVcpWYfvXywsXsgQwbAF0QJm8SSFi0TZm5ykv4WE16afQp08yuZS0 -3U4K3MJCa4rxO58edcxBopWYfQ29K3iINM8enRfr5q+u5mAAbALAEEvyFjgLWl/u -7arxn7bJAgMBAAECggEBAJfMt8KfHzBunrDnVrk8FayYGkfmOzAOkc1yKEx6k/TH -zFB+Mqlm5MaF95P5t3S0J+r36JBAUdEWC38RUNpF9BwMYYGlDxzlsTdCuGYL/q+J -o6NMLXQt7/jQUQqGnWAvPFzqhbcGqOo5R2ZVH25sEWv9PDuRI35XAepIkDTwWsfa -P6UcJJoP+4v9B++fb3sSL4zNwp1BqS4wxR8YTR0t1zQqOxJ5BGPw1J8aBMs1sq5t -qyosAQAT63kLrdqWotHaM26QxjqEQUMlh12XMWb5GdBXUxbvyGtEabsqskGa/f8B -RdHE437J8D8l+jxb2mZLzrlaH3dq2tbFGCe1rT8qLRECgYEA5CWIvoD/YnQydLGA -OlEhCSocqURuqcotg9Ev0nt/C60jkr/NHFLGppz9lhqjIDjixt3sIMGZMFzxRtwM -pSYal3XiR7rZuHau9iM35yDhpuytEiGbYy1ADakJRzY5jq/Qa8RfPP9Atua5xAeP -q6DiSnq9vhHv9G+O4MxzHBmrw9sCgYEAziiJWFthcwvuXn3Jv9xFYKEb/06puZAx -EgQCz/3rPzv5fmGD/sKVo1U/K4z/eA82DNeKG8QRTFJCxT8TCNRxOmGV7HdCYo/B -4BTNNvbKcdi3l0j75kKoADg+nt5CD5lz6gLG0GrUEnVO1y5HVfCTb3BEAfa36C85 -9i0sfQGiwysCgYEAuus9k8cgdct5oz3iLuVVSark/JGCkT2B+OOkaLChsDFUWeEm -7TOsaclpwldkmvvAYOplkZjMJ2GelE2pVo1XcAw3LkmaI5WpVyQXoxe/iQGT8qzy -IFlsh0Scw2lb0tmcyw6CcPk4TiHOxRrkzNrtS9QwLM+JZx0XVHptPPKTVc0CgYAu -j/VFYY5G/8Dc0qhIjyWUR48dQNUQtkJ/ASzpcT46z/7vznKTjbtiYpSb74KbyUO5 -7sygrM4DYOj3x+Eys1jHiNbly6HQxQtS4x/edCsRP5NntfI+9XsgYZOzKhvdjhki -F3J0DEzNxnUCIM+311hVaRPTJbgv1srOkTFlIoNydQKBgQC6/OHGaC/OewQqRlRK -Mg5KZm01/pk4iKrpA5nG7OTAeoa70NzXNtG8J3WnaJ4mWanNwNUOyRMAMrsUAy9q -EeGqHM5mMFpY4TeVuNLL21lu/x3KYw6mKL3Ctinn+JLAoYoqEy8deZnEA5/tjYlz -YhFBchnUicjoUN1chdpM6SpV2Q== +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQDKjrjWZlfOs1Ch +qt1RoyLfqyXbHVXIAW0fTzAxfJnxvFOiWqAAKgC2qVQM8Y080kRUuRaXP/w9ywXT ++MzX6tByy5VbTYJYyTjHOH46EWLNdcqEJs4+FCVqOIYrQPQ6pGAhCXmgBy4Vb42J +ABLwb+Kt+y2Dk15tggVcAHP2Khri+lRXWvda+kZAe2F1IojmuWyCTy3FEYHic5qN +BsXcf6u1oyFV8MybOuz1zGj3vd2C+dEKO4Ohw9rRwnvHSatjM+CfwiXf8kTXzDBF +Z/8W3+6yA49pHxRbG7FE3K1TAPhkrp+BVTIUOcdI74wEA6UEkWFF5sQcmmAth59M +EQrl2CXorftEPhsKZE59dUP1+nYAPvR/mTySNCSw7/rvdf+csRSZ5ollMu/lxsht +ulJYJI03+IiDTn47FI5D+IF25REK7d4LzGIo6T73ktsT+qpSXHuTWC+IABm8AMF9 +7ljxHSwMRU/z+O5uiONRItDAgKH/OItFG54PtY2vAhaO0YiZrZcCAwEAAQKCAYB2 +hTo8IVghlySH5B1p5kXCkDcvVaPaypLaLhCp9Blzq9lX9yUF043lU4Ddrf0RaIsY +88/3IjZqxb+cP0lE0Z20fdDfwqORZfQ2BaU+PuwMAm9EEhy9kDYwR/ChoHkHUyT4 +T7392BWr70Dmt8ddLmp5mK4R/gnTk6+lHJK9p/dhdk4haxWvAyBWHJty2Yk3T6nh +OYkzdUIFidUVza+6jG2hc1lPGv3tmnYKgNeulkblm10oWphz79C6ycx5WG7TNgef +CQ3z7//Nn89YTiaUBjLvoLvxRTMwO96r7E/FaslSl/fWnF3HP3lut26Z/mNfhiwj +qn7AhUwpSNPV0qcxFWXr/rXUjdk745wv8wOODK8atjjE/vt/MRBK0rAOIPSm3ecx +37PKNtR4i+sNeDEcY1IyTHE6wFvJSy5y8AFpn5y8tbqYfhlEVWZ4pcnlrKxhWm7j +oBkB/4GBjKQgbQ7ttym9eNG1wIbZ8v9N06+yeLs/NCc4bFZEgcWjFqBH1bLtAYEC +gcEA8tt8iYNqbsDH2ognjEmbbBxrDBmyYcEKRpg1i1SUopcZl8i93IHpG7EgJIaj +l7aWSbASAxjnK02t0VZ3nNS60acibzRwY/+e8OrSqlQdMXlAB2ggBA86drDJpfBl +WGJG8TJVY9bc1TU2uuwtZR1LAMSsRHVp+3IvKLpHrne5exPd3x6KGYcuaM+Uk/rE +u6tLsFNwaCdh+iBFFDT2bnYIw7jAsokJUkwxMVxSC0/21s2blhO/q5LsN1gFC1kN +TbpXAoHBANWE7TmG2szPvujPwrK18v6iJlHCA2n50AgAQXrsetj2JcF3HYHYdHnq +z36MQ6FpBKOiQumozWvb32WTjEwdG2kix7GEfam4DAUBdqYuCHzPcR12K5Tc8hsX +NG7JXUAeS8ZJEiOdu95X59JHyBxUQtNfte5rcbaV17SVw6K6bsWVJnj60YjtJrpa +xHvv1ZRnT2WEzJGpA+ii1h3I52N7ipGBiw172qcW+bKJukMi8eHxx5CC9e5tBpnu +C+Ou/eYewQKBwHxNa0jXQrq9YY2w8s0TP8HuKbxfyrXOIHxRm9ZczFcMD8VosgUT +WUUbO+B2KXWVtwawYAfFz0ySzcy//SkAmT6F1VIl/QCx7aBSENGti+Ous98WpIxv +XvUxN4T/rl+2raj2ok4fw5g9TG4QRIvkmmciQyonDr/sicbG0bmy/fTJDl8NOpIm +ZtKurNWxHNERtAPkMTyeK7/ilHjrQtb3AzVqcvbuvR6qcONa5YN0wlrfkisWoJwo +707EdpCAXBbUsQKBwQCnpzcpu2Sj+t9ZKIElF87T93gFLETH+ppJHgJMRdDz+NqO +fTwTD2XtsNz57aLQ44f8AFVv6NZbQYq41FEOFrDGLcQE9BZDpDrz10FVnMGXVr7n +tjjkK1SCxwapkr0AsoknCYsPojO4kud46loLPHI4TGeq7HyeNCvqJMo3RRHjXIiX +58GNNUD6hHjRI/FdFH14Jf0GxmJGUU20l2Jwb7nPJJuNm9mE53pqoNA7FP4+Pj1H +kD0Q2FSdmxeE0IuWHEECgcBgw6ogJ/FRRGLcym+aApqP9BChK+W8FDfDc9Mi4p/J +g+XmetWNFGCGTlOefGqUDIkwSG+QVOEN3hxziXbsjnvfpGApqoaulAI5oRvrwIcj +QIvD2mt0PB52k5ZL9QL2K9sgBa43BJDyCKooMAlTy2XMM+NyXVxQKmzf3r3jQ5sl +Rptk7ro38a9G8Rs99RFDyOmP1haOM0KXZvPksN4nsXuTlE01cnwnI29XKAlEZaoA +pQPLXD8W/KK4mwDbmokYXmo= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- -MIIDYjCCAkqgAwIBAgIJALJXRr8qF6oIMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV +MIIEYjCCAsqgAwIBAgIJAJm2YulYpr+6MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u IFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTAeFw0x -ODAxMTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMGIxCzAJBgNVBAYTAlhZMRcwFQYD +ODA4MjkxNDIzMTZaFw0yODA4MjYxNDIzMTZaMGIxCzAJBgNVBAYTAlhZMRcwFQYD VQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZv -dW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTCCASIwDQYJKoZIhvcNAQEB -BQADggEPADCCAQoCggEBALe6VE1+Fs4B7L8H7AhVjH53lrPsRXiuJHrKL+oTob5v -FtO9rLamrJIe0PB65phKHZxWZxqwEoogUmasIPXNHxSYtvmLZcUiKdrW3d548K4F -kyu9lnCTOVBRIzUBm2MY479bQjcXA7rLrjn3EDk924ojtIxiQf9UvfMTlbUnAyi+ -/pu82JP4ZTmXqYH3KUJOAumZSqhCFGrP/fnMLNXKiH6mkO9VylZh+9fLCxeyBDBs -AXRAmbxJIWLRNmbnKS/hYTXpp9CnTzK5lLTdTgrcwkJrivE7nx51zEGilZh9Db0r -eIg0zx6dF+vmr67mYABsAsAQS/IWOAtaX+7tqvGftskCAwEAAaMbMBkwFwYDVR0R -BBAwDoIMZmFrZWhvc3RuYW1lMA0GCSqGSIb3DQEBCwUAA4IBAQCZhHhGItpkqhEq -ntMRd6Hv0GoOJixNvgeMwK4NJSRT/no3OirtUTzccn46h+SWibSa2eVssAV+pAVJ -HbzkN/DH27A1mMx1zJL1ekcOKA1AF6MXhUnrUGXMqW36YNtzHfXJLrwvpLJ13OQg -/Kxo4Nw68bGzM+PyRtKU/mpgYyfcvwR+ZSeIDh1fvUZK/IEVCf8ub42GPVs5wPfv -M+k5aHxWTxeif3K1byTRzxHupYNG2yWO4XEdnBGOuOwzzN4/iQyNcsuQKeuKHGrt -YvIlG/ri04CQ7xISZCj74yjTZ+/A2bXre2mQXAHqKPumHL7cl34+erzbUaxYxbTE -u5FcOmLQ +dW5kYXRpb24xFTATBgNVBAMMDGZha2Vob3N0bmFtZTCCAaIwDQYJKoZIhvcNAQEB +BQADggGPADCCAYoCggGBAMqOuNZmV86zUKGq3VGjIt+rJdsdVcgBbR9PMDF8mfG8 +U6JaoAAqALapVAzxjTzSRFS5Fpc//D3LBdP4zNfq0HLLlVtNgljJOMc4fjoRYs11 +yoQmzj4UJWo4hitA9DqkYCEJeaAHLhVvjYkAEvBv4q37LYOTXm2CBVwAc/YqGuL6 +VFda91r6RkB7YXUiiOa5bIJPLcURgeJzmo0Gxdx/q7WjIVXwzJs67PXMaPe93YL5 +0Qo7g6HD2tHCe8dJq2Mz4J/CJd/yRNfMMEVn/xbf7rIDj2kfFFsbsUTcrVMA+GSu +n4FVMhQ5x0jvjAQDpQSRYUXmxByaYC2Hn0wRCuXYJeit+0Q+GwpkTn11Q/X6dgA+ +9H+ZPJI0JLDv+u91/5yxFJnmiWUy7+XGyG26UlgkjTf4iINOfjsUjkP4gXblEQrt +3gvMYijpPveS2xP6qlJce5NYL4gAGbwAwX3uWPEdLAxFT/P47m6I41Ei0MCAof84 +i0Ubng+1ja8CFo7RiJmtlwIDAQABoxswGTAXBgNVHREEEDAOggxmYWtlaG9zdG5h +bWUwDQYJKoZIhvcNAQELBQADggGBAMIVLp6e6saH2NQSg8iFg8Ewg/K/etI++jHo +gCJ697AY02wtfrBox1XtljlmI2xpJtVAYZWHhrNqwrEG43aB7YEV6RqTcG6QUVqa +NbD8iNCnMKm7fP89hZizmqA1l4aHnieI3ucOqpgooM7FQwLX6qk+rSue6lD5N/5f +avsublnj8rNKyDfHpQ3AWduLoj8QqctpzI3CqoDZNLNzaDnzVWpxT1SKDQ88q7VI +W5zb+lndpdQlCu3v5HM4w5UpwL/k1htl/z6PnPseS2UdlXv6A8KITnCLg5PLP4tz +2oTAg9gjOtRP/0uwkhvicwoFzFJNVT813lzTLE1jlobMPiZhsS1mjaJGPD9GQZDK +ny3j8ogrIRGjnI4xpOMNNDVphcvwtV8fRbvURSHCj9Y4kCLpD5ODuoyEyLYicJIv +GZP456GP0iSCK5GKO0ij/YzGCkPWD5zA+mYFpMMGZPTwajenMw7TVaPXcc9CZBtr +oOjwwiLEqdkpxUj13mJYTlt5wsS/Kw== -----END CERTIFICATE----- diff --git a/Lib/test/keycert3.pem b/Lib/test/keycert3.pem index 621eb08bb0ce..e0a8205a660e 100644 --- a/Lib/test/keycert3.pem +++ b/Lib/test/keycert3.pem @@ -1,64 +1,84 @@ -----BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDgV4G+Zzf2DT5n -oAisIGFhn/bz7Vn5WiXUqbDsxROJOh/7BtOlduZka0pPhFylGbnxS8l1kEWHRI2T -6hOoWzumB6ItKiN+T5J30lAvSyo7iwdFoAQ/S5nPXQfhNARQe/NEOhRtpcuNdyx4 -BWdPdPuJQA1ASNJCLwcLOoRxaLbKLvb2V5T5FCAkeNPtRvPuT4gKQItMmiHfAhoV -C8MZWF/GC0RukHINys5MwqeFexam8CznmQPMYrLdhmKTj3DTivCPoh97EDIFGlgZ -SCaaYDVQA+aqlo/q2pi52PFwC1KzhNEA7EeOqSwC1NQjwjHuhcnf9WbxrgTq2zh3 -rv5YEW2ZAgMBAAECggEAPfSMtTumPcJskIumuXp7yk02EyliZrWZqwBuBwVqHsS5 -nkbFXnXWrLbgn9MrDsFrE5NdgKUmPnQVMVs8sIr5jyGejSCNCs4I4iRn1pfIgwcj -K/xEEALd6GGF0pDd/CgvB5GOoLVf4KKf2kmLvWrOKJpSzoUN5A8+v8AaYYOMr4sC -czbvfGomzEIewEG+Rw9zOVUDlmwyEKPQZ47E7PQ+EEA7oeFdR+1Zj6eT9ndegf8B -54frySYCLRUCk/sHCpWhaJBtBrcpht7Y8CfY7hiH/7x866fvuLnYPz4YALtUb0wN -7zUCNS9ol3n4LbjFFKfZtiRjKaCBRzMjK0rz6ydFcQKBgQDyLI3oGbnW73vqcDe/ -6eR0w++fiCAVhfMs3AO/gOaJi2la2JHlJ5u+cIHQIOFwEhn6Zq0AtdmnFx1TS5IQ -C0MdXI0XoQQw7rEF8EJcvfe85Z0QxENVhzydtdb8QpJfnQGfBfLyQlaaRYzRRHB6 -VdYUHF3EIPVIhbjbghal+Qep/QKBgQDtJlRPHkWwTMevu0J0fYbWN1ywtVTFUR// -k7VyORSf8yuuSnaQRop4cbcqONxmDKH6Or1fl3NYBsAxtXkkOK1E2OZNo2sfQdRa -wpA7o7mPHRhztQFpT5vflp+8P6+PEFat8D04eBOhNwrwwfhiPjD4gv5KvN4XutRW -VWv/2pnmzQKBgHPvHGg2mJ7quvm6ixXW1MWJX1eSBToIjCe3lBvDi5nhIaiZ8Q4w -7gA3QA3xD7tlDwauzLeAVxgEmsdbcCs6GQEfY3QiYy1Bt4FOSZa4YrcNfSmfq1Rw -j3Y4rRjKjeQz96i3YlzToT3tecJc7zPBj+DEy6au2H3Fdn+vQURneWHJAoGBANG7 -XES8mRVaUh/wlM1BVsaNH8SIGfiHzqzRjV7/bGYpQTBbWpAuUrhCmaMVtpXqBjav -TFwGLVRkZAWSYRjPpy2ERenT5SE3rv61o6mbGrifGsj6A82HQmtzYsGx8SmtYXtj -REF0sKebbmmOooUAS379GrguYJzL9o6D7YfRZNrhAoGAVfb/tiFU4S67DSpYpQey -ULhgfsFpDByICY6Potsg67gVFf9jIaB83NPTx3u/r6sHFgxFw7lQsuZcgSuWMu7t -glzOXVIP11Y5sl5CJ5OsfeK1/0umMZF5MWPyAQCx/qrPlZL86vXjt24Y/VaOxsAi -CZYdyJsjgOrJrWoMbo5ta54= +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQCfKC83Qe9/ZGMW +YhbpARRiKco6mJI9CNNeaf7A89TE+w5Y3GSwS8uzqp5C6QebZzPNueg8HYoTwN85 +Z3xM036/Qw9KhQVth+XDAqM+19e5KHkYcxg3d3ZI1HgY170eakaLBvMDN5ULoFOw +Is2PtwM2o9cjd5mfSuWttI6+fCqop8/l8cerG9iX2GH39p3iWwWoTZuYndAA9qYv +07YWajuQ1ESWKPjHYGTnMvu4xIzibC1mXd2M6u/IjNO6g426SKFaRDWQkx01gIV/ +CyKs9DgZoeMHkKZuPqZVOxOK+A/NrmrqHFsPIsrs5wk7QAVju5/X1skpn/UGQlmM +RwBaQULOs1FagA+54RXU6qUPW0YmhJ4xOB4gHHD1vjAKEsRZ7/6zcxMyOm+M1DbK +RTH4NWjVWpnY8XaVGdRhtTpH9MjycpKhF+D2Zdy2tQXtqu2GdcMnUedt13fn9xDu +P4PophE0ip/IMgn+kb4m9e+S+K9lldQl0B+4BcGWAqHelh2KuU0CAwEAAQKCAYEA +lKiWIYjmyRjdLKUGPTES9vWNvNmRjozV0RQ0LcoSbMMLDZkeO0UwyWqOVHUQ8+ib +jIcfEjeNJxI57oZopeHOO5vJhpNlFH+g7ltiW2qERqA1K88lSXm99Bzw6FNqhCRE +K8ub5N9fyfJA+P4o/xm0WK8EXk5yIUV17p/9zJJxzgKgv2jsVTi3QG2OZGvn4Oug +ByomMZEGHkBDzdxz8c/cP1Tlk1RFuwSgews178k2xq7AYSM/s0YmHi7b/RSvptX6 +1v8P8kXNUe4AwTaNyrlvF2lwIadZ8h1hA7tCE2n44b7a7KfhAkwcbr1T59ioYh6P +zxsyPT678uD51dbtD/DXJCcoeeFOb8uzkR2KNcrnQzZpCJnRq4Gp5ybxwsxxuzpr +gz0gbNlhuWtE7EoSzmIK9t+WTS7IM2CvZymd6/OAh1Fuw6AQhSp64XRp3OfMMAAC +Ie2EPtKj4islWGT8VoUjuRYGmdRh4duAH1dkiAXOWA3R7y5a1/y/iE8KE8BtxocB +AoHBAM8aiURgpu1Fs0Oqz6izec7KSLL3l8hmW+MKUOfk/Ybng6FrTFsL5YtzR+Ap +wW4wwWnnIKEc1JLiZ7g8agRETK8hr5PwFXUn/GSWC0SMsazLJToySQS5LOV0tLzK +kJ3jtNU7tnlDGNkCHTHSoVL2T/8t+IkZI/h5Z6wjlYPvU2Iu0nVIXtiG+alv4A6M +Hrh9l5or4mjB6rGnVXeYohLkCm6s/W97ahVxLMcEdbsBo1prm2JqGnSoiR/tEFC/ +QHQnbQKBwQDEu7kW0Yg9sZ89QtYtVQ1YpixFZORaUeRIRLnpEs1w7L1mCbOZ2Lj9 +JHxsH05cYAc7HJfPwwxv3+3aGAIC/dfu4VSwEFtatAzUpzlhzKS5+HQCWB4JUNNU +MQ3+FwK2xQX4Ph8t+OzrFiYcK2g0An5UxWMa2HWIAWUOhnTOydAVsoH6yP31cVm4 +0hxoABCwflaNLNGjRUyfBpLTAcNu/YtcE+KREy7YAAgXXrhRSO4XpLsSXwLnLT7/ +YOkoBWDcTWECgcBPWnSUDZCIQ3efithMZJBciqd2Y2X19Dpq8O31HImD4jtOY0V7 +cUB/wSkeHAGwjd/eCyA2e0x8B2IEdqmMfvr+86JJxekC3dJYXCFvH5WIhsH53YCa +3bT1KlWCLP9ib/g+58VQC0R/Cc9T4sfLePNH7D5ZkZd1wlbV30CPr+i8KwKay6MD +xhvtLx+jk07GE+E9wmjbCMo7TclyrLoVEOlqZMAqshgApT+p9eyCPetwXuDHwa3n +WxhHclcZCV7R4rUCgcAkdGSnxcvpIrDPOUNWwxvmAWTStw9ZbTNP8OxCNCm9cyDl +d4bAS1h8D/a+Uk7C70hnu7Sl2w7C7Eu2zhwRUdhhe3+l4GINPK/j99i6NqGPlGpq +xMlMEJ4YS768BqeKFpg0l85PRoEgTsphDeoROSUPsEPdBZ9BxIBlYKTkbKESZDGR +twzYHljx1n1NCDYPflmrb1KpXn4EOcObNghw2KqqNUUWfOeBPwBA1FxzM4BrAStp +DBINpGS4Dc0mjViVegECgcA3hTtm82XdxQXj9LQmb/E3lKx/7H87XIOeNMmvjYuZ +iS9wKrkF+u42vyoDxcKMCnxP5056wpdST4p56r+SBwVTHcc3lGBSGcMTIfwRXrj3 +thOA2our2n4ouNIsYyTlcsQSzifwmpRmVMRPxl9fYVdEWUgB83FgHT0D9avvZnF9 +t9OccnGJXShAIZIBADhVj/JwG4FbaX42NijD5PNpVLk1Y17OV0I576T9SfaQoBjJ +aH1M/zC4aVaS0DYB/Gxq7v8= -----END PRIVATE KEY----- Certificate: Data: Version: 3 (0x2) Serial Number: - 82:ed:bf:41:c8:80:91:9c - Signature Algorithm: sha1WithRSAEncryption + cb:2d:80:99:5a:69:52:5c + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 19 19:09:06 2018 GMT - Not After : Nov 28 19:09:06 2027 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Jul 7 14:23:16 2028 GMT Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (2048 bit) + Public-Key: (3072 bit) Modulus: - 00:e0:57:81:be:67:37:f6:0d:3e:67:a0:08:ac:20: - 61:61:9f:f6:f3:ed:59:f9:5a:25:d4:a9:b0:ec:c5: - 13:89:3a:1f:fb:06:d3:a5:76:e6:64:6b:4a:4f:84: - 5c:a5:19:b9:f1:4b:c9:75:90:45:87:44:8d:93:ea: - 13:a8:5b:3b:a6:07:a2:2d:2a:23:7e:4f:92:77:d2: - 50:2f:4b:2a:3b:8b:07:45:a0:04:3f:4b:99:cf:5d: - 07:e1:34:04:50:7b:f3:44:3a:14:6d:a5:cb:8d:77: - 2c:78:05:67:4f:74:fb:89:40:0d:40:48:d2:42:2f: - 07:0b:3a:84:71:68:b6:ca:2e:f6:f6:57:94:f9:14: - 20:24:78:d3:ed:46:f3:ee:4f:88:0a:40:8b:4c:9a: - 21:df:02:1a:15:0b:c3:19:58:5f:c6:0b:44:6e:90: - 72:0d:ca:ce:4c:c2:a7:85:7b:16:a6:f0:2c:e7:99: - 03:cc:62:b2:dd:86:62:93:8f:70:d3:8a:f0:8f:a2: - 1f:7b:10:32:05:1a:58:19:48:26:9a:60:35:50:03: - e6:aa:96:8f:ea:da:98:b9:d8:f1:70:0b:52:b3:84: - d1:00:ec:47:8e:a9:2c:02:d4:d4:23:c2:31:ee:85: - c9:df:f5:66:f1:ae:04:ea:db:38:77:ae:fe:58:11: - 6d:99 + 00:9f:28:2f:37:41:ef:7f:64:63:16:62:16:e9:01: + 14:62:29:ca:3a:98:92:3d:08:d3:5e:69:fe:c0:f3: + d4:c4:fb:0e:58:dc:64:b0:4b:cb:b3:aa:9e:42:e9: + 07:9b:67:33:cd:b9:e8:3c:1d:8a:13:c0:df:39:67: + 7c:4c:d3:7e:bf:43:0f:4a:85:05:6d:87:e5:c3:02: + a3:3e:d7:d7:b9:28:79:18:73:18:37:77:76:48:d4: + 78:18:d7:bd:1e:6a:46:8b:06:f3:03:37:95:0b:a0: + 53:b0:22:cd:8f:b7:03:36:a3:d7:23:77:99:9f:4a: + e5:ad:b4:8e:be:7c:2a:a8:a7:cf:e5:f1:c7:ab:1b: + d8:97:d8:61:f7:f6:9d:e2:5b:05:a8:4d:9b:98:9d: + d0:00:f6:a6:2f:d3:b6:16:6a:3b:90:d4:44:96:28: + f8:c7:60:64:e7:32:fb:b8:c4:8c:e2:6c:2d:66:5d: + dd:8c:ea:ef:c8:8c:d3:ba:83:8d:ba:48:a1:5a:44: + 35:90:93:1d:35:80:85:7f:0b:22:ac:f4:38:19:a1: + e3:07:90:a6:6e:3e:a6:55:3b:13:8a:f8:0f:cd:ae: + 6a:ea:1c:5b:0f:22:ca:ec:e7:09:3b:40:05:63:bb: + 9f:d7:d6:c9:29:9f:f5:06:42:59:8c:47:00:5a:41: + 42:ce:b3:51:5a:80:0f:b9:e1:15:d4:ea:a5:0f:5b: + 46:26:84:9e:31:38:1e:20:1c:70:f5:be:30:0a:12: + c4:59:ef:fe:b3:73:13:32:3a:6f:8c:d4:36:ca:45: + 31:f8:35:68:d5:5a:99:d8:f1:76:95:19:d4:61:b5: + 3a:47:f4:c8:f2:72:92:a1:17:e0:f6:65:dc:b6:b5: + 05:ed:aa:ed:86:75:c3:27:51:e7:6d:d7:77:e7:f7: + 10:ee:3f:83:e8:a6:11:34:8a:9f:c8:32:09:fe:91: + be:26:f5:ef:92:f8:af:65:95:d4:25:d0:1f:b8:05: + c1:96:02:a1:de:96:1d:8a:b9:4d Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: @@ -70,11 +90,11 @@ Certificate: X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: - 85:11:BE:16:47:04:D1:30:EE:86:8A:18:70:BE:A8:28:6F:82:3D:CE + 8F:EA:1D:E3:33:5C:00:16:B3:8B:6F:6B:6F:D3:4C:CB:B5:CB:7C:55 X509v3 Authority Key Identifier: - keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48 + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server - serial:82:ED:BF:41:C8:80:91:9B + serial:CB:2D:80:99:5A:69:52:5B Authority Information Access: CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer @@ -85,48 +105,60 @@ Certificate: Full Name: URI:http://testca.pythontest.net/testca/revocation.crl - Signature Algorithm: sha1WithRSAEncryption - 7f:a1:7e:3e:68:01:b0:32:b8:57:b8:03:68:13:13:b3:e3:f4: - 70:2f:15:e5:0f:87:b9:fd:e0:12:e3:16:f2:91:53:c7:4e:25: - af:ca:cb:a7:d9:9d:57:4d:bf:a2:80:d4:78:aa:04:31:fd:6d: - cc:6d:82:43:e9:62:16:0d:0e:26:8b:e7:f1:3d:57:5c:68:02: - 9c:2b:b6:c9:fd:62:2f:10:85:88:cc:44:a5:e7:a2:3e:89:f2: - 1f:02:6a:3f:d0:3c:6c:24:2d:bc:51:62:7a:ec:25:c5:86:87: - 77:35:8f:f9:7e:d0:17:3d:77:56:bf:1a:0c:be:09:78:ee:ea: - 73:97:65:60:94:91:35:b3:5c:46:8a:5e:6d:94:52:de:48:b7: - 1f:6c:28:79:7f:ff:08:8d:e4:7d:d0:b9:0b:7c:ae:c4:1d:2a: - a1:b3:50:11:82:03:5e:6c:e7:26:fa:05:32:39:07:83:49:b9: - a2:fa:04:da:0d:e5:ff:4c:db:97:d0:c3:a7:43:37:4c:16:de: - 3c:b5:e9:7e:82:d4:b3:10:df:d1:c1:66:72:9c:15:67:19:3b: - 7b:91:0a:82:07:67:c5:06:03:5f:80:54:08:81:8a:b1:5c:7c: - 4c:d2:07:38:92:eb:12:f5:71:ae:de:05:15:c8:e1:33:f0:e4: - 96:0f:0f:1e + Signature Algorithm: sha256WithRSAEncryption + 27:f5:8c:59:10:f4:c6:e7:28:00:bf:ba:8d:7b:13:03:f1:1c: + a6:5f:b3:06:55:a4:22:b9:db:b2:d5:46:bd:f7:0c:dd:43:6e: + b4:79:65:67:21:0c:2a:55:ee:40:8e:85:9f:9f:47:bb:0a:2a: + 4d:b6:64:74:98:a0:7f:ae:dc:f1:2e:db:42:77:18:e0:75:8b: + 26:35:68:c3:41:ed:6b:c8:77:72:6f:6a:9a:5d:55:69:02:fd: + 5a:54:c8:57:cb:b0:65:03:16:e2:0f:00:39:99:66:a0:9b:88: + 93:17:e2:5a:2d:79:35:5f:97:57:78:c4:af:f5:99:5e:86:ab: + d3:11:ad:1a:a2:0d:fa:52:10:b9:fe:bf:9d:ce:33:d9:86:b2: + 9c:16:f8:d6:75:08:8a:db:0a:e5:b4:2b:16:7f:b4:f9:2a:9f: + c3:d2:77:d7:cd:65:1e:f4:6c:1e:eb:59:b9:f0:ae:5f:a4:1f: + cc:4a:c4:b9:7a:a9:d9:6b:32:68:3b:e1:65:b0:84:b7:90:c4: + ae:fe:f4:37:4f:21:a0:de:9f:3a:b1:e5:cc:16:04:66:3f:0b: + 41:dc:42:3d:20:3e:ec:b7:95:2b:35:57:fa:be:7f:b6:3a:ba: + ca:4f:58:fe:75:3e:08:89:2c:8c:b0:5d:2e:f9:89:10:2b:f9: + 41:46:4f:3c:00:b7:27:d3:65:24:28:17:23:26:31:42:ea:7e: + 4e:93:e4:7b:68:54:ca:9f:46:f3:ef:2b:e9:85:0c:b5:84:b2: + d5:35:34:80:75:2b:f0:91:23:b8:08:01:8e:b9:0a:54:d4:fb: + 34:52:fe:d9:45:f0:80:3b:b6:c1:6f:82:d1:1f:f2:3b:08:f6: + 46:a6:96:27:61:4b:58:32:7a:0e:1d:59:c5:44:ad:5e:1a:79: + 33:c1:d4:05:2f:4a:d3:d8:42:42:8d:33:e3:63:ca:d5:87:97: + 9b:4d:b8:1a:03:34:bb:1c:d2:02:3f:59:23:e2:23:80:88:63: + c2:f0:a2:63:a8:8b -----BEGIN CERTIFICATE----- -MIIE8TCCA9mgAwIBAgIJAILtv0HIgJGcMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIF8TCCBFmgAwIBAgIJAMstgJlaaVJcMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yNzExMjgx -OTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA3MDcx +NDIzMTZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxv -Y2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOBXgb5nN/YN -PmegCKwgYWGf9vPtWflaJdSpsOzFE4k6H/sG06V25mRrSk+EXKUZufFLyXWQRYdE -jZPqE6hbO6YHoi0qI35PknfSUC9LKjuLB0WgBD9Lmc9dB+E0BFB780Q6FG2ly413 -LHgFZ090+4lADUBI0kIvBws6hHFotsou9vZXlPkUICR40+1G8+5PiApAi0yaId8C -GhULwxlYX8YLRG6Qcg3KzkzCp4V7FqbwLOeZA8xist2GYpOPcNOK8I+iH3sQMgUa -WBlIJppgNVAD5qqWj+ramLnY8XALUrOE0QDsR46pLALU1CPCMe6Fyd/1ZvGuBOrb -OHeu/lgRbZkCAwEAAaOCAcAwggG8MBQGA1UdEQQNMAuCCWxvY2FsaG9zdDAOBgNV -HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud -EwEB/wQCMAAwHQYDVR0OBBYEFIURvhZHBNEw7oaKGHC+qChvgj3OMH0GA1UdIwR2 -MHSAFJrPz27rcT3bPPGuiGtWcgPLCKdIoVGkTzBNMQswCQYDVQQGEwJYWTEmMCQG -A1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNVBAMMDW91 -ci1jYS1zZXJ2ZXKCCQCC7b9ByICRmzCBgwYIKwYBBQUHAQEEdzB1MDwGCCsGAQUF -BzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3RjYS9weWNhY2Vy -dC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQv -dGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly90ZXN0Y2EucHl0 -aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3JsMA0GCSqGSIb3DQEBBQUA -A4IBAQB/oX4+aAGwMrhXuANoExOz4/RwLxXlD4e5/eAS4xbykVPHTiWvysun2Z1X -Tb+igNR4qgQx/W3MbYJD6WIWDQ4mi+fxPVdcaAKcK7bJ/WIvEIWIzESl56I+ifIf -Amo/0DxsJC28UWJ67CXFhod3NY/5ftAXPXdWvxoMvgl47upzl2VglJE1s1xGil5t -lFLeSLcfbCh5f/8IjeR90LkLfK7EHSqhs1ARggNebOcm+gUyOQeDSbmi+gTaDeX/ -TNuX0MOnQzdMFt48tel+gtSzEN/RwWZynBVnGTt7kQqCB2fFBgNfgFQIgYqxXHxM -0gc4kusS9XGu3gUVyOEz8OSWDw8e +Y2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAJ8oLzdB739k +YxZiFukBFGIpyjqYkj0I015p/sDz1MT7DljcZLBLy7OqnkLpB5tnM8256DwdihPA +3zlnfEzTfr9DD0qFBW2H5cMCoz7X17koeRhzGDd3dkjUeBjXvR5qRosG8wM3lQug +U7AizY+3Azaj1yN3mZ9K5a20jr58Kqinz+Xxx6sb2JfYYff2neJbBahNm5id0AD2 +pi/TthZqO5DURJYo+MdgZOcy+7jEjOJsLWZd3Yzq78iM07qDjbpIoVpENZCTHTWA +hX8LIqz0OBmh4weQpm4+plU7E4r4D82uauocWw8iyuznCTtABWO7n9fWySmf9QZC +WYxHAFpBQs6zUVqAD7nhFdTqpQ9bRiaEnjE4HiAccPW+MAoSxFnv/rNzEzI6b4zU +NspFMfg1aNVamdjxdpUZ1GG1Okf0yPJykqEX4PZl3La1Be2q7YZ1wydR523Xd+f3 +EO4/g+imETSKn8gyCf6Rvib175L4r2WV1CXQH7gFwZYCod6WHYq5TQIDAQABo4IB +wDCCAbwwFAYDVR0RBA0wC4IJbG9jYWxob3N0MA4GA1UdDwEB/wQEAwIFoDAdBgNV +HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E +FgQUj+od4zNcABazi29rb9NMy7XLfFUwfQYDVR0jBHYwdIAU3b/K2ubRNLo3dSHK +b5oIKPI1tkihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29m +dHdhcmUgRm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcoIJAMst +gJlaaVJbMIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKGMGh0dHA6Ly90ZXN0 +Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNlcjA1BggrBgEFBQcw +AYYpaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2Evb2NzcC8wQwYD +VR0fBDwwOjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0 +Y2EvcmV2b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQELBQADggGBACf1jFkQ9MbnKAC/ +uo17EwPxHKZfswZVpCK527LVRr33DN1DbrR5ZWchDCpV7kCOhZ+fR7sKKk22ZHSY +oH+u3PEu20J3GOB1iyY1aMNB7WvId3JvappdVWkC/VpUyFfLsGUDFuIPADmZZqCb +iJMX4loteTVfl1d4xK/1mV6Gq9MRrRqiDfpSELn+v53OM9mGspwW+NZ1CIrbCuW0 +KxZ/tPkqn8PSd9fNZR70bB7rWbnwrl+kH8xKxLl6qdlrMmg74WWwhLeQxK7+9DdP +IaDenzqx5cwWBGY/C0HcQj0gPuy3lSs1V/q+f7Y6uspPWP51PgiJLIywXS75iRAr ++UFGTzwAtyfTZSQoFyMmMULqfk6T5HtoVMqfRvPvK+mFDLWEstU1NIB1K/CRI7gI +AY65ClTU+zRS/tlF8IA7tsFvgtEf8jsI9kamlidhS1gyeg4dWcVErV4aeTPB1AUv +StPYQkKNM+NjytWHl5tNuBoDNLsc0gI/WSPiI4CIY8LwomOoiw== -----END CERTIFICATE----- diff --git a/Lib/test/keycert4.pem b/Lib/test/keycert4.pem index b7df7f3f2c71..d1ebb82486de 100644 --- a/Lib/test/keycert4.pem +++ b/Lib/test/keycert4.pem @@ -1,64 +1,84 @@ -----BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDH/76hZAZH4cSV -CmVZa5HEqKCjCKrcPwBECs9BS+3ibwN4x9NnFNP+tCeFGgJXl7WGFoeXgg3oK+1p -FsOWpsRHuF3BdqkCnShSydmT8bLaGHwKeL0cPxJP5T/uW7ezPKW2VWXGMwmwRaRJ -9dj2VCUu20vDZWSGFr9zjnjoJczBtH3RsVUgpK7euEHuQ5pIM9QSOaCo+5FPR7s7 -1nU7YqbFWtd+NhC8Og1G497B31DQlHciF6BRm6/cNGAmHaAErKUGBFdkGtFPHBn4 -vktoEg9fwxJAZLvGpoTZWrB4HRsRwVTmFdGvK+JXK225xF23AXRXp/snhSuSFeLj -E5cpyJJ7AgMBAAECggEAQOv527X2e/sDr0XSpHZQuT/r9UBpBlnFIlFH+fBF5k0X -GWv0ae/O6U1dzs0kmX57xG0n0ry6+vTXeleTYiH8cTOd66EzN9AAOO+hG29IGZf9 -HAEZkkO/FARc/mjzdtFnEYsjIHWM3ZWdwQx3Q28JKu6w51rQiN51g3NqOCGdF/uF -rE5XPKsKndn+nLHvsNuApFgUYZEwdrozgUueEgRaPTUCNhzotcA9eWoBdA24XNhk -x8Cm/bZWabXm7gBO75zl3Cu2F21ay+EuwyOZTsx6lZi6YX9/zo1mkO81Zi3tQk50 -NMEI0feLNwsdxTbmOcVJadjOgd+QVghlFyr5HGBWMQKBgQD3AH3rhnAo6tOyNkGN -+IzIU1MhUS452O7IavykUYO9sM24BVChpRtlI9Dpev4yE/q3BAO3+oWT3cJrN7/3 -iyo1dzAkpGvI65XWfElXFM4nLjEiZzx4W9fiPN91Oucpr0ED6+BZXTtz4gVm0TP/ -TUc2xvTB6EKvIyWmKOYEi0snxQKBgQDPSOjbz9jWOrC9XY7PmtLB6QJDDz7XSGVK -wzD+gDAPpAwhk58BEokdOhBx2Lwl8zMJi0CRHgH2vNvkRyhvUQ4UFzisrqann/Tw -klp5sw3iWC6ERC8z9zL7GfHs7sK3mOVeAdK6ffowPM3JrZ2vPusVBdr0MN3oZwki -CtNXqbY1PwKBgGheQNbAW6wubX0kB9chavtKmhm937Z5v4vYCSC1gOEqUAKt3EAx -L74wwBmn6rjmUE382EVpCgBM99WuHONQXmlxD1qsTw763LlgkuzE0cckcYaD8L06 -saHa7uDuHrcyYlpx1L5t8q0ol/e19i6uTKUMtGcq6OJwC3yGU4sgAIWxAoGBAMVq -qiQXm2vFL+jafxYoXUvDMJ1PmskMsTP4HOR2j8+FrOwZnVk3HxGP6HOVOPRn4JbZ -YiAT1Uj6a+7I+rCyINdvmlGUcTK6fFzW9oZryvBkjcD483/pkktmVWwTpa2YV/Ml -h16IdsyUTGYlDUYHhXtbPUJOfDpIT4F1j/0wrFGfAoGAO82BcUsehEUQE0xvQLIn -7QaFtUI5z19WW730jVuEobiYlh9Ka4DPbKMvka8MwyOxEwhk39gZQavmfG6+wZm+ -kjERU23LhHziJGWS2Um4yIhC7myKbWaLzjHEq72dszLpQku4BzE5fT60fxI7cURD -WGm/Z3Q2weS3ZGIoMj1RNPI= +MIIG/QIBADANBgkqhkiG9w0BAQEFAASCBucwggbjAgEAAoIBgQDGjpiHzq7ghxhM +ZzrnRsGBC/cmw8EREIdbqlrz/l8BFaWeipvO5Hb/MyU8xs2zLUrqIr2JNf+Eii8Y +m4bYmZclFra4jomaiSlxTZOe3dMV8m4vAq4eT2mSfZZC1+XAutqdz7WhHxhMVEm3 +AyTWvTC3qCbnlbX5VIoQUwFrsSWqDiHyaGdK3rrOTKFUKM8YPiq/BZkm6A4eiFci +5wd/SPD+w0pIscZbQW1MUr5bs54uylWaUmtfI8KJt6BDZQ/uA06c6i863sSCEI6L +gq+wyikeJGNMxZMfgu3dzfv4BiZBQX0ZhiRvqseDSdPcuVa2Ifb6CFlg298neweY +4EAIE1O+uqo5h8FF1aUOMZpQEZuzsp9R/TAMBHX1YmVjG/kRdBeaHe3whzB1Pfue +PIX2ZTMmLNYbYbfnmxhk1nn8aAvoT98pNw8y3/2k2KNsu24n9uSkkxAoqJ19WKwm +mL8MpJKAzLv45tRvhN+QLtnRdu+LJ9m29npQHFmYLbdqRfmidnMCAwEAAQKCAYBd +w1C8MRnb5W/QBJ+IP515NxFLOP2e9VM2MkgpGGH8vSAssf/Jv5GCCcD35lmU1zqd +PjKK7PjwueBrmmYfOshpN0Sp+oV4eHUdkCi5yL65inYFtRpMLewIxU2D2zgfvx0l +kMSQhYKP6O22gsGOtmCfGcTlb4kzaHyaINh25nyGxY26TxsX+/3zFbTJbUv+grzk +39vmx4aDXJbpYHfl36gOZmJZ2bl1tnvKovhJjZSRO/MYoPsbPmPLbO89ZCgVmXFc +GVkb5Cram6i3iyutSDjxWN7Fb8uy8pFLPGAXZgF7pQoXPSEHZe8GEWBnWSC9KaDa +uM9Ir847/Muy1ceCmxKcI2WrSjoH2AhPcmHgvbPE9Mynr6+uzReSP3q7Wh9PHm23 +oFx3DwdCfmjysnpAMBawNmJdWyxVDbZ6eyrhp17ADpsMaDTynZ+fJjgMr+MmWtbU +YSRD0wWtl/DrzsaePZsOjCpKYJyulC+rh9/Zz1aiwrGWPbgEAzDrD6Q1Zp0mUCEC +gcEA+XskmGIB9rRPy+YQmRgzQ555PsjLWsnQsNktP6KBhlQjFKJZXRZ0DxDTS7h8 +NrJrUDBmwfsgzggVbeO55VP5FGwD6DNeO/Bz++Fdevh8uKQFHDfk4sbIUPS91qw4 +s7OW7PR7C7Jf7Dnjmsn42o2lO4FsbcEn2F+PHOvoLl/OrSx73lS/RkdOEItW8d8/ +ExRohylnba/I2vCE9bNZd4DGjMW87j/THKPadDZWEqWggcrjY8x6ibSQGm2n2Tka +8B+vAoHBAMu+zl8kqFlYDG24dGfVpMaOYj5Osj0cV5f7O2pZ15sCevjiqoKGHH7G +O8EiI5pRBZ893+Fsx6YWmcKue88lfGvaoQjV0LUbfMfX/FoD39w/ZLx31yKEiFuc +KvMiRV5mO3kQiHBVX9vamzr5NeaErccXY9LnhaKOMX9blgiDQZH7fc3RhodcFWrC +9yfX6ryfidpPnRvK7Ops7hVnFKyyS4FaAarnzH1B2WcVcD4lYYxhMc8VXeU3eKOh +j1fI/F5ifQKBwQDpCjB670HqUzAexL9IYqSwSz3yedoK6m24ZIWx5XicI8fJJIXZ +QHoVAKB/IMtWxH8dnri+Bnj0O/TYe1pQb4pBm0xjAGjMEKYm6LNLhQXr67qiS0vQ +0eKYTKVv+9vTcLRQj2bI3Exh+wkys+tzK9DmrtS8CSvRICIs3+g4OWJzvRPP8NXj +LgQrzBzhPqpKhkvFxdVJTmSOrxFj+a5exLmzEZqT6qanIB+VYpQwQuqVkxGpTX5B +V5ssNLYPYRpapx0CgcByCtQixzcAA1u5knR9pkT76ris3YnA0Ptqk3I3XiBjoGjK +pL0CICUVBMpvmTdKai12a8DDwgqiOaZJJTchxH63NAHNGzkeFkuq5IdYrzB/bHBr +WbzukjZs6KXVv4oKg7ioVAu6rN7iBaO7x8BWzk8i0EHMzFCto1+rRM1e6HEsUBOj +v7LIU0+dmZGUGLRIbhhQPR3Yb6ZatSwyiKc23vmKZqHmUqbQOaqBm6te7beDRugF +XJVY9sqs9IJyhYpVHlUCgcAPoslwYKeAXagsxdQrH3D9VJDXVOHWKMBqQZDio5dB +Q80uWpuxtt6nhZkQO1JIWnYb6v+zbDbcgjByBIDuxCdBW9d+QQnanKmVyrXguK91 +C3OcHHOmSduFdWC3/zYW1mW97Tz1sXyam2hly1u3L5kW+GnE1hr9VVPjQNrO9+Ge +qW0coaJqKY78q3Rm2dtyZeJSFFI1o/DQ3blyItsFpg/QrR+a5XrS6Nw2ZLIL4Azo +J1CTgMwjhwlMNCI4t4dkHd0= -----END PRIVATE KEY----- Certificate: Data: Version: 3 (0x2) Serial Number: - 82:ed:bf:41:c8:80:91:9d - Signature Algorithm: sha1WithRSAEncryption + cb:2d:80:99:5a:69:52:5d + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 19 19:09:06 2018 GMT - Not After : Nov 28 19:09:06 2027 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Jul 7 14:23:16 2028 GMT Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=fakehostname Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (2048 bit) + Public-Key: (3072 bit) Modulus: - 00:c7:ff:be:a1:64:06:47:e1:c4:95:0a:65:59:6b: - 91:c4:a8:a0:a3:08:aa:dc:3f:00:44:0a:cf:41:4b: - ed:e2:6f:03:78:c7:d3:67:14:d3:fe:b4:27:85:1a: - 02:57:97:b5:86:16:87:97:82:0d:e8:2b:ed:69:16: - c3:96:a6:c4:47:b8:5d:c1:76:a9:02:9d:28:52:c9: - d9:93:f1:b2:da:18:7c:0a:78:bd:1c:3f:12:4f:e5: - 3f:ee:5b:b7:b3:3c:a5:b6:55:65:c6:33:09:b0:45: - a4:49:f5:d8:f6:54:25:2e:db:4b:c3:65:64:86:16: - bf:73:8e:78:e8:25:cc:c1:b4:7d:d1:b1:55:20:a4: - ae:de:b8:41:ee:43:9a:48:33:d4:12:39:a0:a8:fb: - 91:4f:47:bb:3b:d6:75:3b:62:a6:c5:5a:d7:7e:36: - 10:bc:3a:0d:46:e3:de:c1:df:50:d0:94:77:22:17: - a0:51:9b:af:dc:34:60:26:1d:a0:04:ac:a5:06:04: - 57:64:1a:d1:4f:1c:19:f8:be:4b:68:12:0f:5f:c3: - 12:40:64:bb:c6:a6:84:d9:5a:b0:78:1d:1b:11:c1: - 54:e6:15:d1:af:2b:e2:57:2b:6d:b9:c4:5d:b7:01: - 74:57:a7:fb:27:85:2b:92:15:e2:e3:13:97:29:c8: - 92:7b + 00:c6:8e:98:87:ce:ae:e0:87:18:4c:67:3a:e7:46: + c1:81:0b:f7:26:c3:c1:11:10:87:5b:aa:5a:f3:fe: + 5f:01:15:a5:9e:8a:9b:ce:e4:76:ff:33:25:3c:c6: + cd:b3:2d:4a:ea:22:bd:89:35:ff:84:8a:2f:18:9b: + 86:d8:99:97:25:16:b6:b8:8e:89:9a:89:29:71:4d: + 93:9e:dd:d3:15:f2:6e:2f:02:ae:1e:4f:69:92:7d: + 96:42:d7:e5:c0:ba:da:9d:cf:b5:a1:1f:18:4c:54: + 49:b7:03:24:d6:bd:30:b7:a8:26:e7:95:b5:f9:54: + 8a:10:53:01:6b:b1:25:aa:0e:21:f2:68:67:4a:de: + ba:ce:4c:a1:54:28:cf:18:3e:2a:bf:05:99:26:e8: + 0e:1e:88:57:22:e7:07:7f:48:f0:fe:c3:4a:48:b1: + c6:5b:41:6d:4c:52:be:5b:b3:9e:2e:ca:55:9a:52: + 6b:5f:23:c2:89:b7:a0:43:65:0f:ee:03:4e:9c:ea: + 2f:3a:de:c4:82:10:8e:8b:82:af:b0:ca:29:1e:24: + 63:4c:c5:93:1f:82:ed:dd:cd:fb:f8:06:26:41:41: + 7d:19:86:24:6f:aa:c7:83:49:d3:dc:b9:56:b6:21: + f6:fa:08:59:60:db:df:27:7b:07:98:e0:40:08:13: + 53:be:ba:aa:39:87:c1:45:d5:a5:0e:31:9a:50:11: + 9b:b3:b2:9f:51:fd:30:0c:04:75:f5:62:65:63:1b: + f9:11:74:17:9a:1d:ed:f0:87:30:75:3d:fb:9e:3c: + 85:f6:65:33:26:2c:d6:1b:61:b7:e7:9b:18:64:d6: + 79:fc:68:0b:e8:4f:df:29:37:0f:32:df:fd:a4:d8: + a3:6c:bb:6e:27:f6:e4:a4:93:10:28:a8:9d:7d:58: + ac:26:98:bf:0c:a4:92:80:cc:bb:f8:e6:d4:6f:84: + df:90:2e:d9:d1:76:ef:8b:27:d9:b6:f6:7a:50:1c: + 59:98:2d:b7:6a:45:f9:a2:76:73 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: @@ -70,11 +90,11 @@ Certificate: X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: - F8:76:79:CB:11:85:F0:46:E5:95:E6:7E:69:CB:12:5E:4E:AA:EC:4D + 52:E0:93:AA:52:55:B7:BB:E7:A8:E0:8C:DE:41:2E:F4:07:F0:36:FB X509v3 Authority Key Identifier: - keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48 + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server - serial:82:ED:BF:41:C8:80:91:9B + serial:CB:2D:80:99:5A:69:52:5B Authority Information Access: CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer @@ -85,48 +105,60 @@ Certificate: Full Name: URI:http://testca.pythontest.net/testca/revocation.crl - Signature Algorithm: sha1WithRSAEncryption - 6d:50:8d:fb:ee:4e:93:8b:eb:47:56:ba:38:cc:80:e1:9d:c7: - e1:9e:1f:9c:22:0c:d2:08:9b:ed:bf:31:d9:00:ee:af:8c:56: - 78:92:d1:7c:ba:4e:81:7f:82:1f:f4:68:99:86:91:c6:cb:57: - d3:b9:41:12:fa:75:53:fd:22:32:21:50:af:6b:4c:b1:34:36: - d1:a8:25:0a:d0:f0:f8:81:7d:69:58:6e:af:e3:d2:c4:32:87: - 79:d7:cd:ad:0c:56:f3:15:27:10:0c:f9:57:59:53:00:ed:af: - 5d:4d:07:86:7a:e5:f3:97:88:bc:86:b4:f1:17:46:33:55:28: - 66:7b:70:d3:a5:12:b9:4f:c7:ed:e6:13:20:2d:f0:9e:ec:17: - 64:cf:fd:13:14:1b:76:ba:64:ac:c5:51:b6:cd:13:0a:93:b1: - fd:43:09:a0:0b:44:6c:77:45:43:0b:e5:ed:70:b2:76:dc:08: - 4a:5b:73:5f:c1:fc:7f:63:70:f8:b9:ca:3c:98:06:5f:fd:98: - d1:e4:e6:61:5f:09:8f:6c:18:86:98:9c:cb:3f:73:7b:3f:38: - f5:a7:09:20:ee:a5:63:1c:ff:8b:a6:d1:8c:e8:f4:84:3d:99: - 38:0f:cc:e0:52:03:f9:18:05:23:76:39:de:52:ce:8e:fb:a6: - 6e:f5:4f:c3 + Signature Algorithm: sha256WithRSAEncryption + 29:d2:3f:82:3f:c1:38:35:a6:bd:81:10:fe:64:ec:ff:7e:e1: + c6:6f:7f:86:65:f9:31:6f:fb:ef:32:4e:2f:87:c8:42:de:6c: + 8d:b8:06:08:8f:37:70:95:7d:e1:40:d4:82:2b:8d:b3:4a:fd: + 34:c5:9e:df:ff:01:53:4a:4f:08:f4:58:e1:74:fc:78:e3:3e: + 71:a7:5e:66:07:ea:d2:04:31:e2:75:a8:4c:80:17:86:92:20: + d2:32:a7:9a:65:8b:1a:5f:f1:4c:c8:50:6d:00:fc:99:bf:69: + b3:48:f3:45:5a:ee:35:50:14:b8:f3:92:92:c6:9f:0e:5d:eb: + 0d:e8:ec:f2:a4:09:6b:dc:66:2b:fc:df:4c:fc:65:a1:ae:d3: + b5:88:6a:a4:e7:08:1c:94:49:e0:b8:c1:04:8c:21:09:6c:55: + 4b:2c:97:10:f1:8c:6c:d0:bb:ba:8d:93:e8:47:8b:4d:8e:7d: + 7d:85:53:18:c8:f8:f4:8f:67:3a:b1:aa:3e:18:34:6c:3a:e6: + a6:c7:2f:be:83:38:f5:d5:e5:d2:17:28:61:6c:b6:49:99:21: + 69:a4:a8:b6:94:76:fd:18:ad:35:52:45:64:fb:f1:5d:8e:bb: + c0:21:2e:59:55:24:af:bb:8f:b2:0a:7b:17:f0:34:97:8e:68: + a9:f2:d0:3e:f6:73:82:f8:7c:4e:9a:70:7d:d6:b3:8c:cc:85: + 04:5c:02:8f:74:da:88:3a:20:a8:7e:c2:9e:b0:dd:56:1f:ce: + cd:42:16:c6:14:91:ad:30:e0:dc:76:f2:2c:56:ea:38:45:d8: + c0:3e:b8:90:fa:f3:38:99:ec:44:07:35:8f:69:62:0c:f9:ef: + b7:9d:e5:15:42:6e:fb:fe:4c:ff:e8:94:5a:1a:b0:80:b2:0e: + 17:3d:e1:87:a8:08:84:93:74:68:8d:29:df:ca:0b:6a:44:32: + 8a:51:3b:d6:38:db:bd:e3:2a:1b:5e:20:48:81:82:19:91:c6: + 87:8c:0f:cd:51:ea -----BEGIN CERTIFICATE----- -MIIE9zCCA9+gAwIBAgIJAILtv0HIgJGdMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIF9zCCBF+gAwIBAgIJAMstgJlaaVJdMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yNzExMjgx -OTA5MDZaMGIxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA3MDcx +NDIzMTZaMGIxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xFTATBgNVBAMMDGZh -a2Vob3N0bmFtZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMf/vqFk -BkfhxJUKZVlrkcSooKMIqtw/AEQKz0FL7eJvA3jH02cU0/60J4UaAleXtYYWh5eC -Degr7WkWw5amxEe4XcF2qQKdKFLJ2ZPxstoYfAp4vRw/Ek/lP+5bt7M8pbZVZcYz -CbBFpEn12PZUJS7bS8NlZIYWv3OOeOglzMG0fdGxVSCkrt64Qe5Dmkgz1BI5oKj7 -kU9HuzvWdTtipsVa1342ELw6DUbj3sHfUNCUdyIXoFGbr9w0YCYdoASspQYEV2Qa -0U8cGfi+S2gSD1/DEkBku8amhNlasHgdGxHBVOYV0a8r4lcrbbnEXbcBdFen+yeF -K5IV4uMTlynIknsCAwEAAaOCAcMwggG/MBcGA1UdEQQQMA6CDGZha2Vob3N0bmFt -ZTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC -MAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFPh2ecsRhfBG5ZXmfmnLEl5OquxNMH0G -A1UdIwR2MHSAFJrPz27rcT3bPPGuiGtWcgPLCKdIoVGkTzBNMQswCQYDVQQGEwJY -WTEmMCQGA1UECgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNV -BAMMDW91ci1jYS1zZXJ2ZXKCCQCC7b9ByICRmzCBgwYIKwYBBQUHAQEEdzB1MDwG -CCsGAQUFBzAChjBodHRwOi8vdGVzdGNhLnB5dGhvbnRlc3QubmV0L3Rlc3RjYS9w -eWNhY2VydC5jZXIwNQYIKwYBBQUHMAGGKWh0dHA6Ly90ZXN0Y2EucHl0aG9udGVz -dC5uZXQvdGVzdGNhL29jc3AvMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly90ZXN0 -Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3Jldm9jYXRpb24uY3JsMA0GCSqGSIb3 -DQEBBQUAA4IBAQBtUI377k6Ti+tHVro4zIDhncfhnh+cIgzSCJvtvzHZAO6vjFZ4 -ktF8uk6Bf4If9GiZhpHGy1fTuUES+nVT/SIyIVCva0yxNDbRqCUK0PD4gX1pWG6v -49LEMod5182tDFbzFScQDPlXWVMA7a9dTQeGeuXzl4i8hrTxF0YzVShme3DTpRK5 -T8ft5hMgLfCe7Bdkz/0TFBt2umSsxVG2zRMKk7H9QwmgC0Rsd0VDC+XtcLJ23AhK -W3Nfwfx/Y3D4uco8mAZf/ZjR5OZhXwmPbBiGmJzLP3N7Pzj1pwkg7qVjHP+LptGM -6PSEPZk4D8zgUgP5GAUjdjneUs6O+6Zu9U/D +a2Vob3N0bmFtZTCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAMaOmIfO +ruCHGExnOudGwYEL9ybDwREQh1uqWvP+XwEVpZ6Km87kdv8zJTzGzbMtSuoivYk1 +/4SKLxibhtiZlyUWtriOiZqJKXFNk57d0xXybi8Crh5PaZJ9lkLX5cC62p3PtaEf +GExUSbcDJNa9MLeoJueVtflUihBTAWuxJaoOIfJoZ0reus5MoVQozxg+Kr8FmSbo +Dh6IVyLnB39I8P7DSkixxltBbUxSvluzni7KVZpSa18jwom3oENlD+4DTpzqLzre +xIIQjouCr7DKKR4kY0zFkx+C7d3N+/gGJkFBfRmGJG+qx4NJ09y5VrYh9voIWWDb +3yd7B5jgQAgTU766qjmHwUXVpQ4xmlARm7Oyn1H9MAwEdfViZWMb+RF0F5od7fCH +MHU9+548hfZlMyYs1htht+ebGGTWefxoC+hP3yk3DzLf/aTYo2y7bif25KSTECio +nX1YrCaYvwykkoDMu/jm1G+E35Au2dF274sn2bb2elAcWZgtt2pF+aJ2cwIDAQAB +o4IBwzCCAb8wFwYDVR0RBBAwDoIMZmFrZWhvc3RuYW1lMA4GA1UdDwEB/wQEAwIF +oDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAd +BgNVHQ4EFgQUUuCTqlJVt7vnqOCM3kEu9AfwNvswfQYDVR0jBHYwdIAU3b/K2ubR +NLo3dSHKb5oIKPI1tkihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRo +b24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZl +coIJAMstgJlaaVJbMIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKGMGh0dHA6 +Ly90ZXN0Y2EucHl0aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNlcjA1Bggr +BgEFBQcwAYYpaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2Evb2Nz +cC8wQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5l +dC90ZXN0Y2EvcmV2b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQELBQADggGBACnSP4I/ +wTg1pr2BEP5k7P9+4cZvf4Zl+TFv++8yTi+HyELebI24BgiPN3CVfeFA1IIrjbNK +/TTFnt//AVNKTwj0WOF0/HjjPnGnXmYH6tIEMeJ1qEyAF4aSINIyp5plixpf8UzI +UG0A/Jm/abNI80Va7jVQFLjzkpLGnw5d6w3o7PKkCWvcZiv830z8ZaGu07WIaqTn +CByUSeC4wQSMIQlsVUsslxDxjGzQu7qNk+hHi02OfX2FUxjI+PSPZzqxqj4YNGw6 +5qbHL76DOPXV5dIXKGFstkmZIWmkqLaUdv0YrTVSRWT78V2Ou8AhLllVJK+7j7IK +exfwNJeOaKny0D72c4L4fE6acH3Ws4zMhQRcAo902og6IKh+wp6w3VYfzs1CFsYU +ka0w4Nx28ixW6jhF2MA+uJD68ziZ7EQHNY9pYgz577ed5RVCbvv+TP/olFoasICy +Dhc94YeoCISTdGiNKd/KC2pEMopRO9Y4273jKhteIEiBghmRxoeMD81R6g== -----END CERTIFICATE----- diff --git a/Lib/test/keycertecc.pem b/Lib/test/keycertecc.pem index deb484f99207..02eda2835f07 100644 --- a/Lib/test/keycertecc.pem +++ b/Lib/test/keycertecc.pem @@ -1,31 +1,31 @@ -----BEGIN PRIVATE KEY----- -MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDDe3QWmhZX07HZbntz4 -CFqAOaoYMdYwD7Z3WPNIc2zR7p4D6BMOa7NAWjLV5A7CUw6hZANiAAQ5IVKzLLz4 -LCfcpy6fMOp+jk5KwywsU3upPtjA6E3UetxPcfnnv+gghRyDAYLN2OVqZgLMEmUo -F1j1SM1QrbhHIuNcVxI9gPPMdumcNFSz/hqxrBRtA/8Z2gywczdNLjc= +MIG2AgEAMBAGByqGSM49AgEGBSuBBAAiBIGeMIGbAgEBBDBL2Y5JfpzbgHw+t4Q+ +c5SHhsZcD9ylEtUMg7OyF9xW6j+3VIVORGaokcOtE0Z2Y5ehZANiAASzz/rInKUz +onpxP5bLxmq8fmrtgRSS0jRPUOU16XKX+KtifnLbmLHQtPrctdkRRROCxnURz2fB +ihQTJkXyBMSswNTRCs+4DUKbMAfihigMVYgdWbZPFBDleo5aeFw4/FM= -----END PRIVATE KEY----- Certificate: Data: Version: 3 (0x2) Serial Number: - 82:ed:bf:41:c8:80:91:9e - Signature Algorithm: sha1WithRSAEncryption + cb:2d:80:99:5a:69:52:5e + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 19 19:09:06 2018 GMT - Not After : Nov 28 19:09:06 2027 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Jul 7 14:23:16 2028 GMT Subject: C=XY, L=Castle Anthrax, O=Python Software Foundation, CN=localhost-ecc Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (384 bit) pub: - 04:39:21:52:b3:2c:bc:f8:2c:27:dc:a7:2e:9f:30: - ea:7e:8e:4e:4a:c3:2c:2c:53:7b:a9:3e:d8:c0:e8: - 4d:d4:7a:dc:4f:71:f9:e7:bf:e8:20:85:1c:83:01: - 82:cd:d8:e5:6a:66:02:cc:12:65:28:17:58:f5:48: - cd:50:ad:b8:47:22:e3:5c:57:12:3d:80:f3:cc:76: - e9:9c:34:54:b3:fe:1a:b1:ac:14:6d:03:ff:19:da: - 0c:b0:73:37:4d:2e:37 + 04:b3:cf:fa:c8:9c:a5:33:a2:7a:71:3f:96:cb:c6: + 6a:bc:7e:6a:ed:81:14:92:d2:34:4f:50:e5:35:e9: + 72:97:f8:ab:62:7e:72:db:98:b1:d0:b4:fa:dc:b5: + d9:11:45:13:82:c6:75:11:cf:67:c1:8a:14:13:26: + 45:f2:04:c4:ac:c0:d4:d1:0a:cf:b8:0d:42:9b:30: + 07:e2:86:28:0c:55:88:1d:59:b6:4f:14:10:e5:7a: + 8e:5a:78:5c:38:fc:53 ASN1 OID: secp384r1 NIST CURVE: P-384 X509v3 extensions: @@ -38,11 +38,11 @@ Certificate: X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: - 33:23:0E:15:04:83:2E:3D:BF:DA:81:6D:10:38:80:C3:C2:B0:A4:74 + C6:82:22:BF:4F:3D:40:AD:9B:16:AD:E7:C5:ED:C4:82:EB:35:97:98 X509v3 Authority Key Identifier: - keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48 + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 DirName:/C=XY/O=Python Software Foundation CA/CN=our-ca-server - serial:82:ED:BF:41:C8:80:91:9B + serial:CB:2D:80:99:5A:69:52:5B Authority Information Access: CA Issuers - URI:http://testca.pythontest.net/testca/pycacert.cer @@ -53,44 +53,54 @@ Certificate: Full Name: URI:http://testca.pythontest.net/testca/revocation.crl - Signature Algorithm: sha1WithRSAEncryption - 3b:6f:97:af:7e:5f:e0:14:34:ed:57:7e:de:ce:c4:85:1e:aa: - 84:52:94:7c:e5:ce:e9:9c:88:8b:ad:b5:4d:16:ac:af:81:ea: - b8:a2:e2:50:2e:cb:e9:11:bd:1b:a6:3f:0c:a2:d7:7b:67:72: - b3:43:16:ad:c6:87:ac:6e:ac:47:78:ef:2f:8c:86:e8:9b:d1: - 43:8c:c1:7a:91:30:e9:14:d6:9f:41:8b:9b:0b:24:9b:78:86: - 11:8a:fc:2b:cd:c9:13:ee:90:4f:14:33:51:a3:c4:9e:d6:06: - 48:f5:41:12:af:f0:f2:71:40:78:f5:96:c2:5d:cf:e1:38:ff: - bf:10:eb:74:2f:c2:23:21:3e:27:f5:f1:f2:af:2c:62:82:31: - 00:c8:96:1b:c3:7e:8d:71:89:e7:40:b5:67:1a:33:fb:c0:8b: - 96:0c:36:78:25:27:82:d8:27:27:52:0f:f7:69:cd:ff:2b:92: - 10:d3:d2:0a:db:65:ed:af:90:eb:db:76:f3:8a:7a:13:9e:c6: - 33:57:15:42:06:13:d6:54:49:fa:84:a7:0e:1d:14:72:ca:19: - 8e:2b:aa:a4:02:54:3c:f6:1c:23:81:7a:59:54:b0:92:65:72: - c8:e5:ba:9f:03:4e:30:f2:4d:45:85:e3:35:a8:b1:68:58:b9: - 3b:20:a3:eb + Signature Algorithm: sha256WithRSAEncryption + 76:e3:19:4d:34:78:50:3e:fa:63:53:d6:3f:01:87:e8:f4:a3: + a9:81:5b:31:d6:de:3a:98:f3:bb:70:4d:29:35:1f:b0:6a:b3: + 9d:bf:03:2b:79:c4:f2:0b:32:f8:fc:f6:cb:13:47:28:81:fa: + 96:b3:1a:1d:bd:4b:f6:35:df:87:ef:6e:74:63:87:3d:7e:2b: + c6:78:d4:8e:ef:03:e6:01:11:22:4e:1b:ef:2c:c1:c5:4e:3f: + 4a:07:ae:92:ef:d3:ac:79:59:7c:60:89:4b:3d:39:08:ef:c4: + 9a:dc:b0:8b:ee:5f:30:40:d3:c2:f3:f8:90:77:9d:8c:a7:07: + b9:5f:62:83:4d:37:fa:36:e1:1d:26:2b:cc:8f:7c:6f:f1:23: + 87:71:48:40:ad:6b:30:16:47:4c:d7:98:bb:f5:9b:63:c8:66: + 47:65:58:d2:c1:07:81:14:0c:25:20:87:b9:1d:ab:0b:56:db: + 2c:ab:36:db:7f:c7:42:52:af:91:d6:fb:18:cf:94:f7:1e:25: + 99:ce:20:78:c6:f8:69:6e:9c:53:f3:fe:90:3e:4d:ca:d5:d6: + ac:6e:02:17:be:4a:0f:fe:e6:14:d4:ce:25:df:17:8f:6f:b9: + d3:28:dc:b4:98:ef:05:6f:eb:20:14:1c:c1:e9:9d:02:7b:0e: + 0f:e4:a8:bc:3b:62:e0:42:0c:b0:f7:a1:63:fe:98:d7:aa:b0: + f6:ed:ff:ab:4f:1a:9a:8f:eb:f0:86:61:d2:d3:a5:08:d0:db: + e4:d6:a9:0e:ec:08:6f:af:fb:ef:73:3f:47:69:97:90:b2:5a: + 6f:31:66:a7:4c:32:0c:e9:ea:18:ce:a9:79:9c:f5:c4:42:f5: + 68:53:b2:a4:8c:98:3f:97:34:62:61:41:0a:54:d7:0b:cd:33: + c8:62:62:da:f7:07:c6:c6:3b:fa:68:ca:5f:62:3e:57:db:bd: + cb:16:94:07:9a:b5:31:55:b8:f8:cb:b0:7f:a0:d1:82:df:71: + c8:90:60:b3:88:b0 -----BEGIN CERTIFICATE----- -MIIESzCCAzOgAwIBAgIJAILtv0HIgJGeMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIEyzCCAzOgAwIBAgIJAMstgJlaaVJeMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yNzExMjgx -OTA5MDZaMGMxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA3MDcx +NDIzMTZaMGMxCzAJBgNVBAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEj MCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24xFjAUBgNVBAMMDWxv -Y2FsaG9zdC1lY2MwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQ5IVKzLLz4LCfcpy6f -MOp+jk5KwywsU3upPtjA6E3UetxPcfnnv+gghRyDAYLN2OVqZgLMEmUoF1j1SM1Q -rbhHIuNcVxI9gPPMdumcNFSz/hqxrBRtA/8Z2gywczdNLjejggHEMIIBwDAYBgNV +Y2FsaG9zdC1lY2MwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASzz/rInKUzonpxP5bL +xmq8fmrtgRSS0jRPUOU16XKX+KtifnLbmLHQtPrctdkRRROCxnURz2fBihQTJkXy +BMSswNTRCs+4DUKbMAfihigMVYgdWbZPFBDleo5aeFw4/FOjggHEMIIBwDAYBgNV HREEETAPgg1sb2NhbGhvc3QtZWNjMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAU -BggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUMyMO -FQSDLj2/2oFtEDiAw8KwpHQwfQYDVR0jBHYwdIAUms/PbutxPds88a6Ia1ZyA8sI -p0ihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg -Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcoIJAILtv0HIgJGb +BggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUxoIi +v089QK2bFq3nxe3Egus1l5gwfQYDVR0jBHYwdIAU3b/K2ubRNLo3dSHKb5oIKPI1 +tkihUaRPME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg +Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcoIJAMstgJlaaVJb MIGDBggrBgEFBQcBAQR3MHUwPAYIKwYBBQUHMAKGMGh0dHA6Ly90ZXN0Y2EucHl0 aG9udGVzdC5uZXQvdGVzdGNhL3B5Y2FjZXJ0LmNlcjA1BggrBgEFBQcwAYYpaHR0 cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2Evb2NzcC8wQwYDVR0fBDww OjA4oDagNIYyaHR0cDovL3Rlc3RjYS5weXRob250ZXN0Lm5ldC90ZXN0Y2EvcmV2 -b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQEFBQADggEBADtvl69+X+AUNO1Xft7OxIUe -qoRSlHzlzumciIuttU0WrK+B6rii4lAuy+kRvRumPwyi13tncrNDFq3Gh6xurEd4 -7y+Mhuib0UOMwXqRMOkU1p9Bi5sLJJt4hhGK/CvNyRPukE8UM1GjxJ7WBkj1QRKv -8PJxQHj1lsJdz+E4/78Q63QvwiMhPif18fKvLGKCMQDIlhvDfo1xiedAtWcaM/vA -i5YMNnglJ4LYJydSD/dpzf8rkhDT0grbZe2vkOvbdvOKehOexjNXFUIGE9ZUSfqE -pw4dFHLKGY4rqqQCVDz2HCOBellUsJJlcsjlup8DTjDyTUWF4zWosWhYuTsgo+s= +b2NhdGlvbi5jcmwwDQYJKoZIhvcNAQELBQADggGBAHbjGU00eFA++mNT1j8Bh+j0 +o6mBWzHW3jqY87twTSk1H7Bqs52/Ayt5xPILMvj89ssTRyiB+pazGh29S/Y134fv +bnRjhz1+K8Z41I7vA+YBESJOG+8swcVOP0oHrpLv06x5WXxgiUs9OQjvxJrcsIvu +XzBA08Lz+JB3nYynB7lfYoNNN/o24R0mK8yPfG/xI4dxSECtazAWR0zXmLv1m2PI +ZkdlWNLBB4EUDCUgh7kdqwtW2yyrNtt/x0JSr5HW+xjPlPceJZnOIHjG+GlunFPz +/pA+TcrV1qxuAhe+Sg/+5hTUziXfF49vudMo3LSY7wVv6yAUHMHpnQJ7Dg/kqLw7 +YuBCDLD3oWP+mNeqsPbt/6tPGpqP6/CGYdLTpQjQ2+TWqQ7sCG+v++9zP0dpl5Cy +Wm8xZqdMMgzp6hjOqXmc9cRC9WhTsqSMmD+XNGJhQQpU1wvNM8hiYtr3B8bGO/po +yl9iPlfbvcsWlAeatTFVuPjLsH+g0YLfcciQYLOIsA== -----END CERTIFICATE----- diff --git a/Lib/test/make_ssl_certs.py b/Lib/test/make_ssl_certs.py index b908c40c11ef..362276583f50 100644 --- a/Lib/test/make_ssl_certs.py +++ b/Lib/test/make_ssl_certs.py @@ -69,7 +69,7 @@ dir = cadir database = $dir/index.txt crlnumber = $dir/crl.txt - default_md = sha1 + default_md = sha256 default_days = 3600 default_crl_days = 3600 certificate = pycacert.pem @@ -108,7 +108,7 @@ def make_cert_key(hostname, sign=False, extra_san='', - ext='req_x509_extensions_full', key='rsa:2048'): + ext='req_x509_extensions_full', key='rsa:3072'): print("creating cert for " + hostname) tempnames = [] for i in range(3): @@ -174,7 +174,7 @@ def make_ca(): t.flush() with tempfile.NamedTemporaryFile() as f: args = ['req', '-new', '-days', '3650', '-extensions', 'v3_ca', '-nodes', - '-newkey', 'rsa:2048', '-keyout', 'pycakey.pem', + '-newkey', 'rsa:3072', '-keyout', 'pycakey.pem', '-out', f.name, '-subj', '/C=XY/L=Castle Anthrax/O=Python Software Foundation CA/CN=our-ca-server'] check_call(['openssl'] + args) diff --git a/Lib/test/pycacert.pem b/Lib/test/pycacert.pem index 850fa32aef7e..73150c960f35 100644 --- a/Lib/test/pycacert.pem +++ b/Lib/test/pycacert.pem @@ -2,78 +2,98 @@ Certificate: Data: Version: 3 (0x2) Serial Number: - 82:ed:bf:41:c8:80:91:9b - Signature Algorithm: sha1WithRSAEncryption + cb:2d:80:99:5a:69:52:5b + Signature Algorithm: sha256WithRSAEncryption Issuer: C=XY, O=Python Software Foundation CA, CN=our-ca-server Validity - Not Before: Jan 19 19:09:06 2018 GMT - Not After : Jan 17 19:09:06 2028 GMT + Not Before: Aug 29 14:23:16 2018 GMT + Not After : Aug 26 14:23:16 2028 GMT Subject: C=XY, O=Python Software Foundation CA, CN=our-ca-server Subject Public Key Info: Public Key Algorithm: rsaEncryption - Public-Key: (2048 bit) + Public-Key: (3072 bit) Modulus: - 00:c3:18:69:6b:c9:47:29:98:8e:b1:56:c2:2e:fa: - 0e:5e:bc:23:80:b3:07:62:24:d2:42:5b:f1:4a:bf: - a9:c8:21:75:c8:e3:e6:2c:1f:87:3c:6e:7c:1b:ed: - 39:32:95:b7:40:b2:60:48:c3:9a:16:08:fe:6d:67: - 88:34:3b:77:77:70:1c:70:5a:d1:1f:5f:04:21:54: - b9:0c:e3:41:85:1d:58:ee:2f:ed:f3:0e:ef:d8:23: - a1:fa:73:fb:4c:28:e0:e5:e6:4d:0b:02:52:49:86: - c7:be:7e:bd:e6:56:76:8b:70:8e:0a:8f:06:33:20: - 1d:7b:5b:aa:d0:c5:1b:ab:9b:cc:54:09:3c:bf:e4: - 40:66:f1:fb:d6:f7:16:9d:c4:19:d4:c3:f2:ff:07: - bc:6f:5a:9e:25:1b:02:4a:a5:ec:42:96:3a:70:d2: - 6c:99:2b:ce:be:e8:d2:01:ef:d5:ba:b0:cf:94:3e: - 82:d0:01:d6:4b:71:80:03:0a:12:45:86:79:81:d8: - 4b:d2:e8:b5:b7:2c:6c:9a:4c:8a:10:10:e4:e4:f5: - df:ce:84:91:ca:d1:46:e0:84:73:17:66:db:69:43: - 78:80:83:be:14:4d:f1:3e:1a:d6:6c:f5:de:45:f3: - 39:af:91:d5:3d:54:44:bf:41:cc:73:68:1a:fc:24: - db:91 + 00:97:ed:55:41:ba:36:17:95:db:71:1c:d3:e1:61: + ac:58:73:e3:c6:96:cf:2b:1f:b8:08:f5:9d:4b:4b: + c7:30:f6:b8:0b:b3:52:72:a0:bb:c9:4d:3b:8e:df: + 22:8e:01:57:81:c9:92:73:cc:00:c6:ec:70:b0:3a: + 17:40:c1:df:f2:8c:36:4c:c4:a7:81:e7:b6:24:68: + e2:a0:7e:35:07:2f:a0:5b:f9:45:46:f7:1e:f0:46: + 11:fe:ca:1a:3c:50:f1:26:a9:5f:9c:22:9c:f8:41: + e1:df:4f:12:95:19:2f:5c:90:01:17:6e:7e:3e:7d: + cf:e9:09:af:25:f8:f8:42:77:2d:6d:5f:36:f2:78: + 1e:7d:4a:87:68:63:6c:06:71:1b:8d:fa:25:fe:d4: + d3:f5:a5:17:b1:ef:ea:17:cb:54:c8:27:99:80:cb: + 3c:45:f1:2c:52:1c:dd:1f:51:45:20:50:1e:5e:ab: + 57:73:1b:41:78:96:de:84:a4:7a:dd:8f:30:85:36: + 58:79:76:a0:d2:61:c8:1b:a9:94:99:63:c6:ee:f8: + 14:bf:b4:52:56:31:97:fa:eb:ac:53:9e:95:ce:4c: + c4:5a:4a:b7:ca:03:27:5b:35:57:ce:02:dc:ec:ca: + 69:f8:8a:5a:39:cb:16:20:15:03:24:61:6c:f4:7a: + fc:b6:48:e5:59:10:5c:49:d0:23:9f:fb:71:5e:3a: + e9:68:9f:34:72:80:27:b6:3f:4c:b1:d9:db:63:7f: + 67:68:4a:6e:11:f8:e8:c0:f4:5a:16:39:53:0b:68: + de:77:fa:45:e7:f8:91:cd:78:cd:28:94:97:71:54: + fb:cf:f0:37:de:c9:26:c5:dc:1b:9e:89:6d:09:ac: + c8:44:71:cb:6d:f1:97:31:d5:4c:20:33:bf:75:4a: + a0:e0:dc:69:11:ed:2a:b4:64:10:11:30:8b:0e:b0: + a7:10:d8:8a:c5:aa:1b:c8:26:8a:25:e7:66:9f:a5: + 6a:1a:2f:7c:5f:83:c6:78:4f:1f Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: - 9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48 + DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 X509v3 Authority Key Identifier: - keyid:9A:CF:CF:6E:EB:71:3D:DB:3C:F1:AE:88:6B:56:72:03:CB:08:A7:48 + keyid:DD:BF:CA:DA:E6:D1:34:BA:37:75:21:CA:6F:9A:08:28:F2:35:B6:48 X509v3 Basic Constraints: CA:TRUE - Signature Algorithm: sha1WithRSAEncryption - 10:25:c8:dc:0c:55:5c:cb:83:6e:79:ef:77:ec:0d:8e:0c:06: - c1:4b:0c:d6:f7:75:52:21:b8:17:4a:38:88:9d:b3:78:c4:42: - fb:b8:7c:14:38:10:fb:ac:da:11:00:5b:42:87:5e:45:9f:6d: - 4e:42:a4:9a:18:06:39:0f:45:a6:96:89:32:d6:59:b3:d3:8e: - e3:95:b6:c4:a2:4b:74:2f:67:c1:fb:bb:f9:72:6f:37:4a:e7: - f4:48:33:71:df:b8:f5:e6:41:3f:d5:d5:2f:26:09:f8:0e:92: - ff:70:ea:f6:ab:58:fb:90:04:d6:43:2e:8f:b1:fb:06:ab:69: - d0:dc:a8:f8:5b:07:f2:d4:66:1f:63:f8:5d:c1:9e:41:44:bb: - c9:e8:7d:e0:46:e4:a7:c8:32:5f:31:62:e5:1c:5c:89:dd:b7: - a2:4f:9e:0d:13:b8:5f:b1:84:53:4c:1f:ce:19:e1:01:00:5e: - bf:41:55:94:a9:a5:13:db:f2:59:f3:d6:4e:b9:9d:9d:b9:0a: - d9:b2:18:6d:7c:b1:f7:96:aa:bd:f6:f9:95:0f:4a:6e:3c:7c: - 46:5b:df:d4:78:ec:9a:dc:e2:e3:01:e6:88:77:39:93:9c:ba: - 2a:63:f9:25:4b:4f:ac:08:79:39:c6:7b:df:07:35:ba:c0:c2: - 50:bf:5a:81 + Signature Algorithm: sha256WithRSAEncryption + 33:6a:54:d3:6b:c0:d7:01:5f:9d:f4:05:c1:93:66:90:50:d0: + b7:18:e9:b0:1e:4a:a0:b6:da:76:93:af:84:db:ad:15:54:31: + 15:13:e4:de:7e:4e:0c:d5:09:1c:34:35:b6:e5:4c:d6:6f:65: + 7d:32:5f:eb:fc:a9:6b:07:f7:49:82:e5:81:7e:07:80:9a:63: + f8:2c:c3:40:bc:8f:d4:2a:da:3e:d1:ee:08:b7:4d:a7:84:ca: + f4:3f:a1:98:45:be:b1:05:69:e7:df:d7:99:ab:1b:ee:8b:30: + cc:f7:fc:e7:d4:0b:17:ae:97:bf:e4:7b:fd:0f:a7:b4:85:79: + e3:59:e2:16:87:bf:1f:29:45:2c:23:93:76:be:c0:87:1d:de: + ec:2b:42:6a:e5:bb:c8:f4:0a:4a:08:0a:8c:5c:d8:7d:4d:d1: + b8:bf:d5:f7:29:ed:92:d1:94:04:e8:35:06:57:7f:2c:23:97: + 87:a5:35:8d:26:d3:1a:47:f2:16:d7:d9:c6:d4:1f:23:43:d3: + 26:99:39:ca:20:f4:71:23:6f:0c:4a:76:76:f7:76:1f:b3:fe: + bf:47:b0:fc:2a:56:81:e1:d2:dd:ee:08:d8:f4:ff:5a:dc:25: + 61:8a:91:02:b9:86:1c:f2:50:73:76:25:35:fc:b6:25:26:15: + cb:eb:c4:2b:61:0c:1c:e7:ee:2f:17:9b:ec:f0:d4:a1:84:e7: + d2:af:de:e4:1b:24:14:a7:01:87:e3:ab:29:58:46:a0:d9:c0: + 0a:e0:8d:d7:59:d3:1b:f8:54:20:3e:78:a5:a5:c8:4f:8b:03: + c4:96:9f:ec:fb:47:cf:76:2d:8d:65:34:27:bf:fa:ae:01:05: + 8a:f3:92:0a:dd:89:6c:97:a1:c7:e7:60:51:e7:ac:eb:4b:7d: + 2c:b8:65:c9:fe:5d:6a:48:55:8e:e4:c7:f9:6a:40:e1:b8:64: + 45:e9:b5:59:29:a5:5f:cf:7d:58:7d:64:79:e5:a4:09:ac:1e: + 76:65:3d:94:c4:68 -----BEGIN CERTIFICATE----- -MIIDbTCCAlWgAwIBAgIJAILtv0HIgJGbMA0GCSqGSIb3DQEBBQUAME0xCzAJBgNV +MIIEbTCCAtWgAwIBAgIJAMstgJlaaVJbMA0GCSqGSIb3DQEBCwUAME0xCzAJBgNV BAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUgRm91bmRhdGlvbiBDQTEW -MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODAxMTkxOTA5MDZaFw0yODAxMTcx -OTA5MDZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg -Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCASIwDQYJKoZI -hvcNAQEBBQADggEPADCCAQoCggEBAMMYaWvJRymYjrFWwi76Dl68I4CzB2Ik0kJb -8Uq/qcghdcjj5iwfhzxufBvtOTKVt0CyYEjDmhYI/m1niDQ7d3dwHHBa0R9fBCFU -uQzjQYUdWO4v7fMO79gjofpz+0wo4OXmTQsCUkmGx75+veZWdotwjgqPBjMgHXtb -qtDFG6ubzFQJPL/kQGbx+9b3Fp3EGdTD8v8HvG9aniUbAkql7EKWOnDSbJkrzr7o -0gHv1bqwz5Q+gtAB1ktxgAMKEkWGeYHYS9LotbcsbJpMihAQ5OT1386EkcrRRuCE -cxdm22lDeICDvhRN8T4a1mz13kXzOa+R1T1URL9BzHNoGvwk25ECAwEAAaNQME4w -HQYDVR0OBBYEFJrPz27rcT3bPPGuiGtWcgPLCKdIMB8GA1UdIwQYMBaAFJrPz27r -cT3bPPGuiGtWcgPLCKdIMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADggEB -ABAlyNwMVVzLg25573fsDY4MBsFLDNb3dVIhuBdKOIids3jEQvu4fBQ4EPus2hEA -W0KHXkWfbU5CpJoYBjkPRaaWiTLWWbPTjuOVtsSiS3QvZ8H7u/lybzdK5/RIM3Hf -uPXmQT/V1S8mCfgOkv9w6varWPuQBNZDLo+x+waradDcqPhbB/LUZh9j+F3BnkFE -u8nofeBG5KfIMl8xYuUcXIndt6JPng0TuF+xhFNMH84Z4QEAXr9BVZSppRPb8lnz -1k65nZ25CtmyGG18sfeWqr32+ZUPSm48fEZb39R47Jrc4uMB5oh3OZOcuipj+SVL -T6wIeTnGe98HNbrAwlC/WoE= +MBQGA1UEAwwNb3VyLWNhLXNlcnZlcjAeFw0xODA4MjkxNDIzMTZaFw0yODA4MjYx +NDIzMTZaME0xCzAJBgNVBAYTAlhZMSYwJAYDVQQKDB1QeXRob24gU29mdHdhcmUg +Rm91bmRhdGlvbiBDQTEWMBQGA1UEAwwNb3VyLWNhLXNlcnZlcjCCAaIwDQYJKoZI +hvcNAQEBBQADggGPADCCAYoCggGBAJftVUG6NheV23Ec0+FhrFhz48aWzysfuAj1 +nUtLxzD2uAuzUnKgu8lNO47fIo4BV4HJknPMAMbscLA6F0DB3/KMNkzEp4HntiRo +4qB+NQcvoFv5RUb3HvBGEf7KGjxQ8SapX5winPhB4d9PEpUZL1yQARdufj59z+kJ +ryX4+EJ3LW1fNvJ4Hn1Kh2hjbAZxG436Jf7U0/WlF7Hv6hfLVMgnmYDLPEXxLFIc +3R9RRSBQHl6rV3MbQXiW3oSket2PMIU2WHl2oNJhyBuplJljxu74FL+0UlYxl/rr +rFOelc5MxFpKt8oDJ1s1V84C3OzKafiKWjnLFiAVAyRhbPR6/LZI5VkQXEnQI5/7 +cV466WifNHKAJ7Y/TLHZ22N/Z2hKbhH46MD0WhY5Uwto3nf6Ref4kc14zSiUl3FU ++8/wN97JJsXcG56JbQmsyERxy23xlzHVTCAzv3VKoODcaRHtKrRkEBEwiw6wpxDY +isWqG8gmiiXnZp+lahovfF+DxnhPHwIDAQABo1AwTjAdBgNVHQ4EFgQU3b/K2ubR +NLo3dSHKb5oIKPI1tkgwHwYDVR0jBBgwFoAU3b/K2ubRNLo3dSHKb5oIKPI1tkgw +DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAYEAM2pU02vA1wFfnfQFwZNm +kFDQtxjpsB5KoLbadpOvhNutFVQxFRPk3n5ODNUJHDQ1tuVM1m9lfTJf6/ypawf3 +SYLlgX4HgJpj+CzDQLyP1CraPtHuCLdNp4TK9D+hmEW+sQVp59/Xmasb7oswzPf8 +59QLF66Xv+R7/Q+ntIV541niFoe/HylFLCOTdr7Ahx3e7CtCauW7yPQKSggKjFzY +fU3RuL/V9yntktGUBOg1Bld/LCOXh6U1jSbTGkfyFtfZxtQfI0PTJpk5yiD0cSNv +DEp2dvd2H7P+v0ew/CpWgeHS3e4I2PT/WtwlYYqRArmGHPJQc3YlNfy2JSYVy+vE +K2EMHOfuLxeb7PDUoYTn0q/e5BskFKcBh+OrKVhGoNnACuCN11nTG/hUID54paXI +T4sDxJaf7PtHz3YtjWU0J7/6rgEFivOSCt2JbJehx+dgUees60t9LLhlyf5dakhV +juTH+WpA4bhkRem1WSmlX899WH1keeWkCawedmU9lMRo -----END CERTIFICATE----- diff --git a/Lib/test/pycakey.pem b/Lib/test/pycakey.pem index 16b75879d09b..c283f8909868 100644 --- a/Lib/test/pycakey.pem +++ b/Lib/test/pycakey.pem @@ -1,28 +1,40 @@ -----BEGIN PRIVATE KEY----- -MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDDGGlryUcpmI6x -VsIu+g5evCOAswdiJNJCW/FKv6nIIXXI4+YsH4c8bnwb7TkylbdAsmBIw5oWCP5t -Z4g0O3d3cBxwWtEfXwQhVLkM40GFHVjuL+3zDu/YI6H6c/tMKODl5k0LAlJJhse+ -fr3mVnaLcI4KjwYzIB17W6rQxRurm8xUCTy/5EBm8fvW9xadxBnUw/L/B7xvWp4l -GwJKpexCljpw0myZK86+6NIB79W6sM+UPoLQAdZLcYADChJFhnmB2EvS6LW3LGya -TIoQEOTk9d/OhJHK0UbghHMXZttpQ3iAg74UTfE+GtZs9d5F8zmvkdU9VES/Qcxz -aBr8JNuRAgMBAAECggEAZHgv4hg3k45C/cSmH7caq2LMDb0kskAwH4hlzI7DipLg -q2Hh6Rsbc92aAG+8IvbC9ohl2VMSCQL8s667j9qH/XQ40QuT4kn2QIv2+FIYLcsd -Pxxjt+YbUf2XrvkHkwMCPqLJTkAVzFOijdGLThF83vZJz9oz4SRKyno8j2LSix68 -WEfnjdyWqYb0eS0luKrLHw+IL7bD5vfc/P0q6u31zJ9h8zEyN5EBCj5OxM/hD0VO -nObrp6r9Bs+xx+yRx+8J5Db6LPXggl5nBqsqrDKVDe6uTysYVgstqkfaDv1L78Vu -3BNdKPAdJ+ucPJrQufzFHBDIIN+Xwckf/09gdQagGQKBgQDnvFaOjZfqc6wL/kNK -tszQtedbdwP20L+EWdNEVsVWK1TOw36Pmkrp2AYLXMd7W1QQu0KukM89EFb84wKo -s4C9V/ch162mUhEAveaLioi7bMwMPIib2V6pHmYGG8nQVRvgkZVYx6ZtPEvWye1v -wmCzzxxK0gC6PQGxp8MSv9yXDwKBgQDXhe57ufc52pgJ+Agyl4PLkllIbG2DKQHG -LwY06v73jllirTpWBOBvN0NvEsI2Pj4aK/BXRNYN1PS7xi/3C6MVWxnOpBtbq3H5 -DwFb5mpfgJmhV6DZ6jMw7h3Yvy35ViKoiI9UK3eTmhkerH3DsILEje7jE9dGmIOJ -4oLa50JjXwKBgQDdTfyveMNasIrejTzATmC89Or0a22KuQIdKBddjSw5xXnhV8s2 -4temCJqFIV6UDLz0mZDt2vc+zqr0KOtyJrLMoAQv+qQoUPlR5wkTvAImU5luGiUw -CN+gzJoMPV93KMBNr1qcBVaHvWyDvCWXdF8beLABOBpfwUEr4xWlgzrruwKBgCvf -cr2zDJW1Xu/gkuKhn02ofA5XLC/gACF03yGUmNSSILYKp25tTba2HD8XJXvfTcsM -GL/bHmvwZuV2obr7nnYxdl5vX7ZYfzoBCPjJPew1BJEognD50PPr9R1zRYuVMjb2 -nZ63vn7IhsaMvIlCfExAzFljZ5ZSY6yE9LhVDVmnAoGBALOwMwpkm1drx5UNSJO7 -70Q8kYzg0oQhCo/7b6DWbAglDPSWQS5IA4rHYOwL3sE+69G2Exe+1454rVDisojW -XdSyA3svI/YQeom8R2LIM/ayCPxCc3/Dxy9+aQQT4lW3F0XQIxod/QsQJxpZIOnF -jOSPclypgV2X6dDOwDkd2Tgh +MIIG/AIBADANBgkqhkiG9w0BAQEFAASCBuYwggbiAgEAAoIBgQCX7VVBujYXldtx +HNPhYaxYc+PGls8rH7gI9Z1LS8cw9rgLs1JyoLvJTTuO3yKOAVeByZJzzADG7HCw +OhdAwd/yjDZMxKeB57YkaOKgfjUHL6Bb+UVG9x7wRhH+yho8UPEmqV+cIpz4QeHf +TxKVGS9ckAEXbn4+fc/pCa8l+PhCdy1tXzbyeB59SodoY2wGcRuN+iX+1NP1pRex +7+oXy1TIJ5mAyzxF8SxSHN0fUUUgUB5eq1dzG0F4lt6EpHrdjzCFNlh5dqDSYcgb +qZSZY8bu+BS/tFJWMZf666xTnpXOTMRaSrfKAydbNVfOAtzsymn4ilo5yxYgFQMk +YWz0evy2SOVZEFxJ0COf+3FeOulonzRygCe2P0yx2dtjf2doSm4R+OjA9FoWOVML +aN53+kXn+JHNeM0olJdxVPvP8DfeySbF3BueiW0JrMhEcctt8Zcx1UwgM791SqDg +3GkR7Sq0ZBARMIsOsKcQ2IrFqhvIJool52afpWoaL3xfg8Z4Tx8CAwEAAQKCAYB6 +1g1kwyYRE70FS4WUaOBr8+dqHW0LNO4bcFrpEi/PSuurqiUzQPoT3DoXXhoWLseN +zGh476yBKZJDKfS7CwYCmZMdprK4uZvu/E6f7Or7EGrbckOtCQkew8iw9L8ZnWgd +FjyThPjdUIdLgidIHcDJWjVHuLKh3B9KD+ZpEU/IjYtRLvbCPJSKQMQShrBE1Rau +SF6IF5P4vK7X0162NlQqMLpQBAKLml93VJcERzVY1u53JJnkG1loIrNvE32zvZ0C +ZnGVpWwamixVrO9K66F+Ml3Y3bkguF8aPUitc+l+yPmUSXmcDcKmhw9DZA0k5t39 +FxVYGn1uJlvHll8QvV9aZtzuTSkAN8VWNufGwivLLsoRlRb1LtGWqHY583FlUWtz +99ABCBehZ2EpAP+MrRj+pyKuMrkQH61bbOhjqifBM8MhHdn9uNmMpvnKyGPMIdRY +cLH4i2S5aQVvmsQbOa98DLOFGXdf+z5HuwdxHtG1S3J7jkT+FkIyYDehJA3X8UEC +gcEAyCpD8rMFfR5qLwrajhy8vbV7TIkNfFHEkm9tCWDBHwuOJqA0DFuMDAKl7cMv +Uo7Z6R2Fqe2OyWvxYkOi/CSjvtT+PTiA0ux1tXUZcxcRSIsLqQZV+elsKcv+QJPy +vf02vNvHjaMaRwl+NYtqpfr1s/3EdJnWCNC3nV1dD+mWVJoO3kGAK5grLAPM1/uP +stARN5Tnh3Doh8e1Yl4V4UKcVqyVqDykX1OLSmPqNH86T4Um0B4h+jf4UBBdDBz1 +rD3JAoHBAMJOZ3M7LqX+F2haSrF/hnG1y9qAqDWGsvy+8YgjFwPFWu7LvqLuXLuz +S3+5GGhplMuM0itqA9PyPotlgtG5O9hAU8SyMitrx1uTW+Q2U3iYPZQ9O327l1cO +F2jKljq0aJrXp+5iWUq8t/FG6DAqYYUCY/X1SheqEXCgCh4ldRhXig3TBYbVZNs9 +7azN0lk408AO/Hx7WYreFQVS7A/EJhk/M1yyIqnJESuxkDefjV4hTVkRXhh+MrCe +vF/jHqh5pwKBwHxXPQRbzvINXbrBTEjxcxGJ1gESNg1fIfQxQZOMxgrJ+9DkvdBb +YiDn2DldgV0Qni8ghrKrfoKDClyXVXy6KfnWh+Rx4BymhOxmxJto3fSpY2HpLKll +JirErLli7my1CjbBdDH4+s7cB8mtRF+9CLp5znr8QSgSt60KnU/QM/F0Df5kxADQ +syjRZ4NXoslaVQeo+TZ6nggSuAtWFNNstH9nEESE/zq0RBe+/3MDAa76MMUhosuz +zw21TIfEyZvoeQKBwDpszNpvPzWWU3+DNtZsXAaw/Vz0Np/xorwwxfuDYZY2r4MC +LI5dUfD2losPIvGyXZVfAIshU4lVW80adt2M7xu1K/sHAeLgg49bndPfKfYnAM0k +JFFIKNd6WzudPtLkEFgO5GXfmK3KVRztjz98XtpZv6jjWqYG8zuEQ8aQyMbK+63w +d8b1P2BVHLRLJybA2Zr0ZqMfi+sfn/570pNjDXml8VG8FoQq+0jCGXVAOofFR7ay +bDK9L4zADjBe4IcUHQKBwFwj8TEVlWxtG+IWO5d+vyuP0OPjSYOmYq4dCMyZ2+Xy +Y+XDYEhlgGTVxafRMTwt57VV3hJTtRxUZziDA++atr8+gPio+QHBYg1JgCKsqKYL +TGoSVrM1jTfdl1orwkpgQmq2q5j7ExpNA3Spsm5kyCaJ1S/8Ivusqaod8S4t7UhW +BRec3gQ+UYv/V3Pc9hS1Zdzf5+G+PDOYoNmveY16hcu0DKc/PlqGtJuBoQjjH7ir +1YVK9GAgLk0VqJvePnF57A== -----END PRIVATE KEY----- diff --git a/Lib/test/revocation.crl b/Lib/test/revocation.crl index 53cb4b3721d9..c05461ca7f93 100644 --- a/Lib/test/revocation.crl +++ b/Lib/test/revocation.crl @@ -1,11 +1,14 @@ -----BEGIN X509 CRL----- -MIIBpjCBjwIBATANBgkqhkiG9w0BAQUFADBNMQswCQYDVQQGEwJYWTEmMCQGA1UE +MIICJjCBjwIBATANBgkqhkiG9w0BAQsFADBNMQswCQYDVQQGEwJYWTEmMCQGA1UE CgwdUHl0aG9uIFNvZnR3YXJlIEZvdW5kYXRpb24gQ0ExFjAUBgNVBAMMDW91ci1j -YS1zZXJ2ZXIXDTE4MDExOTE5MDkwNloXDTI3MTEyODE5MDkwNlqgDjAMMAoGA1Ud -FAQDAgEAMA0GCSqGSIb3DQEBBQUAA4IBAQBYVzH8n2LdyJJ/t8CpWjz652hZJ0sY -DeNYcwuTPzR9CbSwEwNbf0kY9bgWXGfoRD2SnnCnuNNJXO2MuXtxf6qYx2ZjhJm8 -qgxXs0Bz4agRswNMbumjHCmqIv1t88vbrO0+ItEZDK7RJVIMBtVJ0XYOHvD/IG/i -zqa1Fl3uCTvQbTJ2TrqzJeP/Vl40hOD+VdBBZK3j0r4AkCKU3tAiHYTGmHKhPxy1 -f8Yet+4SRMGp1BdDezTI1bICpSZhRJ4geW0UzuCZnXPW8IZzioUmdUBAmAMHPWFr -B0sTTc/ntD4jHG1/T5b0oiDMbXIbh5Iz9iQNcY0IbotkCw39h+K90wY6 +YS1zZXJ2ZXIXDTE4MDgyOTE0MjMxNloXDTI4MDcwNzE0MjMxNlqgDjAMMAoGA1Ud +FAQDAgEAMA0GCSqGSIb3DQEBCwUAA4IBgQCPhrtGSbuvxPAI3YWQFDB4iOWdBnVk +ugW1lsifmCsE86FfID0EwUut1SRHlksltMtcoULMEIdu8yMLWci++4ve22EEuMKT +HUc3T/wBIuQUhA7U4deFG8CZPAxRpNoK470y7dkD4OVf0Gxa6WYDl9z8mXKmWCB9 +hvzqVfLWNSLTAVPsHtkD5PXdi5yRkQr6wYD7poWaIvkpsn7EKCY6Tw5V3rsbRuZq +AGVCq5TH3mctcmwLloCJ4Xr/1q0DsRrYxeeLYxE+UpvvCbVBKgtjBK7zINS7AbcJ +CYCYKUwGWv1fYKJ+KQQHf75mT3jQ9lWuzOj/YWK4k1EBnYmVGuKKt73lLFxC6h3y +MUnaBZc1KZSyJj0IxfHg/o6qx8NgKOl9XRIQ5g5B30cwpPOskGhEhodbTTY3bPtm +RQ36JvQZngzmkhyhr+MDEV5yUTOShfUiclzQOx26CmLmLHWxOZgXtFZob/oKrvbm +Gen/+7K7YTw6hfY52U7J2FuQRGOyzBXfBYQ= -----END X509 CRL----- diff --git a/Lib/test/ssl_cert.pem b/Lib/test/ssl_cert.pem index b1dd3f387f7c..de596717bd85 100644 --- a/Lib/test/ssl_cert.pem +++ b/Lib/test/ssl_cert.pem @@ -1,20 +1,26 @@ -----BEGIN CERTIFICATE----- -MIIDWTCCAkGgAwIBAgIJAPm6B21bar2bMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV +MIIEWTCCAsGgAwIBAgIJAJinz4jHSjLtMA0GCSqGSIb3DQEBCwUAMF8xCzAJBgNV BAYTAlhZMRcwFQYDVQQHDA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9u -IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODAx -MTkxOTA5MDZaFw0yODAxMTcxOTA5MDZaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH +IFNvZnR3YXJlIEZvdW5kYXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xODA4 +MjkxNDIzMTVaFw0yODA4MjYxNDIzMTVaMF8xCzAJBgNVBAYTAlhZMRcwFQYDVQQH DA5DYXN0bGUgQW50aHJheDEjMCEGA1UECgwaUHl0aG9uIFNvZnR3YXJlIEZvdW5k -YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEP -ADCCAQoCggEBAKvvsX2gEti4shve3iYMc+jE4Se7WHs1Bol2f21H8qNboDOFdeb1 -RKHjmq3exHpajywOUEgne9nKHJY/3f2phR4Y5klqG6liLgiSpVyRlcBGbeT2qEAj -9oLiLFUXLGfGDds2mTwivQDLJBWi51j7ff5k2Pr58fN5ugYMn24T9FNyn0moT+qj -SFoBNm58l9jrdkJSlgWfqPlbiMa+mqDn/SFtrwLF2Trbfzu42Sd9UdIzMaSSrzbN -sGm53pNhCh8KndWUQ8GPP2IsLPoUU4qAtmZuTxCx2S1cXrN9EkmT69tlOH84YfSn -96Ih9bWRc7M5y5bfVdEVM+fKQl3hBRf05qMCAwEAAaMYMBYwFAYDVR0RBA0wC4IJ -bG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQAtQ8f37cCEk7/rAcbYR53ce3iK -Vpihb0U2ni1QjG9Tg9UIExkIGkwTiCm7kwQL+GEStBu9AG/QVrOjeTriRiddhWkk -ze8kRaI3AC/63t6Vh9Q1x6PESgeE4OtAO9JpJCf4GILglA789Y/b/GF8zJZQxR13 -qpB4ZwWw7gCBhdEW59u6CFeBmfDa58hM8lWvuVoRrTi7bjUeC6PAn5HVMzZSykhu -4HaUfBp6bKFjuym2+h/VvM1n8C3chjVSmutsLb6ELdD8IK0vPV/yf5+LN256zSsS -dyUZYd8XwQaioEMKdbhLvnehyzHiWfQIUR3BdhONxoIJhHv/EAo8eCkHHYIF +YXRpb24xEjAQBgNVBAMMCWxvY2FsaG9zdDCCAaIwDQYJKoZIhvcNAQEBBQADggGP +ADCCAYoCggGBALKUqUtopT6E68kN+uJNEt34i2EbmG/bwjcD8IaMsgJPSsMO2Bpd +3S6qWgkCeOyCfmAwBxK2kNbxGb63ouysEv7l8GCTJTWv3hG/HQcejJpnAEGi6K1U +fDbyE/db6yZ12SoHVTGkadN4vYGCPd1Wj9ZO1F877SHQ8rDWX3xgTWkxN2ojBw44 +T8RHSDiG8D/CvG4uEy+VUszL+Uvny5y2poNSqvI3J56sptWSrh8nIIbkPZPBdUne +LYMOHTFK3ZjXSmhlXgziTxK71nnzM3Y9K9gxPnRqoXbvu/wFo55hQCkETiRkYgmm +jXcBMZ0TClQVnQWuLjMthRnWFZs4Lfmwqjs7FZD/61581R2BYehvpWbLvvuOJhwv +DFzexL2sXcAl7SsxbzeQKRHqGbIDfbnQTXfs3/VC6Ye5P82P2ucj+XC32N9piRmO +gCBP8L3ub+YzzdxikZN2gZXXE2jsb3QyE/R2LkWdWyshpKe+RsZP1SBRbHShUyOh +yJ90baoiEwj2mwIDAQABoxgwFjAUBgNVHREEDTALgglsb2NhbGhvc3QwDQYJKoZI +hvcNAQELBQADggGBAHRUO/UIHl3jXQENewYayHxkIx8t7nu40iO2DXbicSijz5bo +5//xAB6RxhBAlsDBehgQP1uoZg+WJW+nHu3CIVOU3qZNZRaozxiCl2UFKcNqLOmx +R3NKpo1jYf4REQIeG8Yw9+hSWLRbshNteP6bKUUf+vanhg9+axyOEOH/iOQvgk/m +b8wA8wNa4ujWljPbTQnj7ry8RqhTM0GcAN5LSdSvcKcpzLcs3aYwh+Z8e30sQWna +F40sa5u7izgBTOrwpcDm/w5kC46vpRQ5fnbshVw6pne2by0mdMECASid/p25N103 +jMqTFlmO7kpf/jpCSmamp3/JSEE1BJKHwQ6Ql4nzRA2N1mnvWH7Zxcv043gkHeAu +0x8evpvwuhdIyproejNFlBpKmW8OX7yKTCPPMC/VkX8Q1rVkxU0DQ6hmvwZlhoKa +9Wc2uXpw9xF8itV4Uvcdr3dwqByvIqn7iI/gB+4l41e0u8OmH2MKOx4Nxlly5TNW +HcVKQHyOeyvnINuBAQ== -----END CERTIFICATE----- diff --git a/Lib/test/ssl_key.passwd.pem b/Lib/test/ssl_key.passwd.pem index 669c7ce3c815..e4f1370ab270 100644 --- a/Lib/test/ssl_key.passwd.pem +++ b/Lib/test/ssl_key.passwd.pem @@ -1,30 +1,42 @@ -----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED -DEK-Info: DES-EDE3-CBC,2D5DD30B9D440DBB +DEK-Info: DES-EDE3-CBC,8064BE1494B24B13 -01gIwpy3XxPGsY0PoK59vAxdLhkVj3odO0Z1ULamUzIte6ThKL1HqnZiUlXpYKfK -XqHVVeQ1xouxiDRNFLJ4CqBG4HbRtqTkl+sfaNTVveL18lOOMAZy6W3dCGAnWOTZ -Z0RJyZlQxxjNQLuko4tIvFkrShXIgdiVFjwAhRU0KTUb7UQ2xfFA9R0Kfde30pzz -zSjb/OmYqAIhkdvafGXvJxzZAorQkU9akDh+uJ6cht5B/RGZsbKACYDSv2WSV5yW -r+fKVYcTup33r0Jj8hAD6fVY15K8BJknpkF9HfSlZnmmr2WDaffLokOOnCV/I1ie -WD7ENA7K//48km5D3Ogh2b2/0Iwuzjq8Mvd8aR39N9nINbGR+HNT85pawoo1S0W9 -pQTU4XTmxfXjtR2287C6XZyQ/tBwvNDMFPVhlxsGOdLYwoV5e/L1t1qIfkTlbuvd -JaMzOhSSLjiC156IFoH7PTPe+g75hw2b32XJURFGlaYknHF7P4BmCiwXOQYo5CCo -MQGGlw5qBCqODrIsc03wpL2jUzgvyPqLyaw395ITuSoGX+WO7vUQaGW0Tz/sOoTs -3pK+bTi2QmqZMe7xBOj07CYMMOo4QPrM6NpbObt+Jja2UXaxvKa9BwqCEQzA4pQZ -8ZHHfEWIaDffKTGkAlqm+S8qCtsrEZJhIn3aI/ikzK8v+YkWw6w+8t/tR1V8ET/s -CoYGIR7I8WhdfKAwgx2QT5bt1jkYKJyKPm4Iacp2mNh9gNFVq+JSKF318e7BrR3+ -wyqMkDxRYnov3ybtf6kiICxPREDqa6UG1xRq3SbWz6NnIF/1hoHs79YlSYbMfXNU -ffIxBaXNCcH6jM9duP2YRnO29jLwfLM/mmokTBBjyOBaKZia9GPa4naXoATXW3z+ -Xx4EKIUkKdb53kiV6NtEKMPialAnkeoHTEjyLPgaV8mCHLvGQbnxbYwvpPJH0e2f -CWgiw6ci4ROOzcZ7HJHIDUprwK0xRKn43hoI44fivlSHOFX6da6o3wIqhEUqMKwL -JQDS1GORRk1ndRXP+7Ub1dO+Vo/DqO1VcTr2o5RwZ1LWPnzLqbCG50mvTLH4djB+ -+hf6vlmnFC30N3yUFXWE5vS10nJHYP88dD9CB2RsaWzpxD9Zxl+PKcRsppen6HyO -u3b71a/TBOkJcI+lkOatEFvbuqzBAqhMceMctO+Dl55RFsbxfIw/IXZjdP0PYZ0C -t20DrIdBsvl9F/mfYpmkV4DF7yci78DqnRBcxylVNF2vwX7o+2fq/TsEwsHn3KnT -kvcF5Cq8Vr5C8ugWX8JfveNym0BjLu6Lr58qS4a6qCNGEGPFKyB+xkm4KEScbarQ -aLbEbfulMM7q9//sEOOLexIx7mNoLd29Xzn5hsLCAZLWX6wMq6JVJ/zbBOAHDbBT -yhi03yd5Kvw3swSt4QZj+uR3qTFwxkXUFiVvrSfxRZoyKsxsLr9Z7D8aoH9Rkb2L -6KjZ31nt9Drh7NJfh6ReANBW6INdDW0Y2mbzoDozLszAYjVfuUUEE76iJqXY0N4W -kNr0OQQTUtDpVk0AZZZvy17xV+rkqGgwlOqTvHbwFYEQvgwVz4EKUw== +KJrffOMbo8M0I3PzcYxRZGMpKD1yB3Ii4+bT5XoanxjIJ+4fdx6LfZ0Rsx+riyzs +tymsQu/iYY9j+4rCvN9+eetsL1X6iZpiimKsLexcid9M3fb0vxED5Sgw0dvunCUA +xhqjLIKR92MKbODHf6KrDKCpsiPbjq4gZ7P+uCGXAMHL3MXIJSC0hW9rK7Ce6oyO +CjpIcgB8x+GUWZZZhAFdlzIHMZrteNP2P5HK6QcaT71P034Dz1hhqoj4Q0t+Fta2 +4tfsM/bnTR/l6hwlhPa1e3Uj322tDTDWBScgWANn5+sEWldLmozMaWhZsn22pfk2 +KjRMGXG024JVheV882nbdOBvG7oq+lxkZ/ZP+vvqJqnvYtf7WtM8UivzYpe5Hz5b +kVvWzPjBLUSZ9whM9rDLqSSqMPyPvDTuEmLkuq+xm7pYJmsLqIMP2klZLqRxLX6K +uqwplb8UG440qauxgnQ905PId1l2fJEnRtV+7vXprA0L0QotgXLVHBhLmTFM+3PH +9H3onf31dionUAPrn3nfVE36HhvVgRyvDBnBzJSIMighgq21Qx/d1dk0DRYi1hUI +nCHl0YJPXheVcXR7JiSF2XQCAaFuS1Mr7NCXfWZOZQC/0dkvmHnl9DUAhuqq9BNZ +1cKhZXcKHadg2/r0Zup/oDzmHPUEfTAXT0xbqoWlhkdwbF2veWQ96A/ncx3ISTb4 +PkXBlX9rdia8nmtyQDQRn4NuvchbaGkj4WKFC8pF8Hn7naHqwjpHaDUimBc0CoQW +edNJqruKWwtSVLuwKHCC2gZFX9AXSKJXJz/QRSUlhFGOhuF/J6yKaXj6n5lxWNiQ +54J+OP/hz2aS95CD2+Zf1SKpxdWiLZSIQqESpmmUrXROixNJZ/Z7gI74Dd9dSJOH +W+3AU03vrrFZVrJVZhjcINHoH1Skh6JKscH18L6x4U868nSr4SrRLX8BhHllOQyD +bmU+PZAjF8ZBIaCtTGulDXD29F73MeAZeTSsgQjFu0iKLj1wPiphbx8i/SUtR4YP +X6PVA04g66r1NBw+3RQASVorZ3g1MSFvITHXcbKkBDeJH2z1+c6t/VVyTONnQhM5 +lLgRSk6HCbetvT9PKxWrWutA12pdBYEHdZhMHVf2+xclky7l09w8hg2/qqcdGRGe +oAOZ72t0l5ObNyaruDKUS6f4AjOyWq/Xj5xuFtf1n3tQHyslSyCTPcAbQhDfTHUx +vixb/V9qvYPt7OCn8py7v1M69NH42QVFAvwveDIFjZdqfIKBoJK2V4qPoevJI6uj +Q5ByMt8OXOjSXNpHXpYQWUiWeCwOEBXJX8rzCHdMtg37jJ0zCmeErR1NTdg+EujM +TWYgd06jlT67tURST0aB2kg4ijKgUJefD313LW1zC6gVsTbjSZxYyRbPfSP6flQB +yCi1C19E2OsgleqbkBVC5GlYUzaJT7SGjCRmGx1eqtbrALu+LVH24Wceexlpjydl ++s2nf/DZlKun/tlPh6YioifPCJjByZMQOCEfIox6BkemZETz8uYA4TTWimG13Z03 +gyDGC2jdpEW414J2qcQDvrdUgJ+HlhrAAHaWpMQDbXYxBGoZ+3+ORvQV4kAsCwL8 +k3EIrVpePdik+1xgOWsyLj6QxFXlTMvL6Wc5pnArFPORsgHEolJvxSPTf9aAHNPn +V2WBvxiLBtYpGrujAUM40Syx/aN2RPtcXYPAusHUBw+S8/p+/8Kg8GZmnIXG3F89 +45Eepl2quZYIrou7a1fwIpIIZ0hFiBQ1mlHVMFtxwVHS1bQb3SU2GeO+JcGjdVXc +04qeGuQ5M164eQ5C0T7ZQ1ULiUlFWKD30m+cjqmZzt3d7Q0mKpMKuESIuZJo/wpD +Nas432aLKUhcNx/pOYLkKJRpGZKOupQoD5iUj/j44o8JoFkDK33v2S57XB5QGz28 +9Zuhx49b3W8mbM6EBanlQKLWJGCxXqc/jhYhFWn+b0MhidynFgA0oeWvf6ZDyt6H +Yi5Etxsar09xp0Do3NxtQXLuSUu0ji2pQzSIKuoqQWKqldm6VrpwojiqJhy4WQBQ +aVVyFeWBC7G3Zj76dO+yp2sfJ0itJUQ8AIB9Cg0f34rEZu+r9luPmqBoUeL95Tk7 +YvCOU3Jl8Iqysv8aNpVXT8sa8rrSbruWCByEePZ37RIdHLMVBwVY0eVaFQjrjU7E +mXmM9eaoYLfXOllsQ+M2+qPFUITr/GU3Qig13DhK/+yC1R6V2a0l0WRhMltIPYKW +Ztvvr4hK5LcYCeS113BLiMbDIMMZZYGDZGMdC8DnnVbT2loF0Rfmp80Af31KmMQ4 +6XvMatW9UDjBoY5a/YMpdm7SRwm+MgV2KNPpc2kST87/yi9oprGAb8qiarHiHTM0 -----END RSA PRIVATE KEY----- diff --git a/Lib/test/ssl_key.pem b/Lib/test/ssl_key.pem index b63f38bc5cf2..1ea4578d81ec 100644 --- a/Lib/test/ssl_key.pem +++ b/Lib/test/ssl_key.pem @@ -1,28 +1,40 @@ -----BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCr77F9oBLYuLIb -3t4mDHPoxOEnu1h7NQaJdn9tR/KjW6AzhXXm9USh45qt3sR6Wo8sDlBIJ3vZyhyW -P939qYUeGOZJahupYi4IkqVckZXARm3k9qhAI/aC4ixVFyxnxg3bNpk8Ir0AyyQV -oudY+33+ZNj6+fHzeboGDJ9uE/RTcp9JqE/qo0haATZufJfY63ZCUpYFn6j5W4jG -vpqg5/0hba8Cxdk62387uNknfVHSMzGkkq82zbBpud6TYQofCp3VlEPBjz9iLCz6 -FFOKgLZmbk8QsdktXF6zfRJJk+vbZTh/OGH0p/eiIfW1kXOzOcuW31XRFTPnykJd -4QUX9OajAgMBAAECggEAHppmXDbuw9Z0FVPg9KLIysioTtsgz6VLiZIm8juZK4x2 -glUh/D7xvWL2uDXrgN+3lh7iGUW13LkFx5SMncbbo9TIwI57Z/XKvcnkVwquve+L -RfLFVc1Q5lD9lROv2rS86KTaN4LzYz3FKXi6dvMkpPAsUtfEQhMLkmISypQQq/1z -EJaqo7r85OjN7e0wKazlKZpOzJEa5FQLMVRjTRFhLFNbHXX/tAet2jw+umATKbw8 -hYgiuZ44TwSEd9JeIV/oSYWfI/3HetuYW0ru3caiztRF2NySNu8lcsWgNC7fIku9 -mcHjtSNzs91QN1Qlu7GQvvhpt6OWDirNDCW+49WGaQKBgQDg9SDhfF0jRYslgYbH -cqO4ggaFdHjrAAYpwnAgvanhFZL/zEqm5G1E7l/e2fCkJ9VOSFO0A208chvwMcr+ -dCjHE2tVdE81aQ2v/Eo83VdS1RcOV4Y75yPH48rMhxPaHvxWD/FFDbf0/P2mtPB7 -SZ3kIeZMkE1wxdaO3AKUbQoozwKBgQDDqYgg7kVtygyICE1mB8Hwp6nUxFTczG7y -4XcsDqMIrKmw+PbQluvkoHoStxeVrsTloDhkTjIrpmYLyAiazg+PUJdkd6xrfLSj -VV6X93W0S/1egEb1F1CGFxtk8v/PWH4K76EPL2vxXdxjywz3GWlrL9yDYaB2szzS -DqgwVMqx7QKBgDCD7UF0Bsoyl13RX3XoPXLvZ+SkR+e2q52Z94C4JskKVBeiwX7Y -yNAS8M4pBoMArDoj0xmBm69rlKbqtjLGbnzwrTdSzDpim7cWnBQgUFLm7gAD1Elb -AhZ8BCK0Bw4FnLoa2hfga4oEfdfUMgEE0W5/+SEOBgWKRUmuHUhRc911AoGAY2EN -YmSDYSM5wDIvVb5k9B3EtevOiqNPSw/XnsoEZtiEC/44JnQxdltIBY93bDBrk5IQ -cmoBM4h91kgQjshQwOMXMhFSwvmBKmCm/hrTbvMVytTutXfVD3ZXFKwT4DW7N0TF -ElhsxBh/YzRz7mG62JVjtFt2zDN3ld2Z8YpvtXUCgYEA4EJ4ObS5YyvcXAKHJFo6 -Fxmavyrf8LSm3MFA65uSnFvWukMVqqRMReQc5jvpxHKCis+XvnHzyOfL0gW9ZTi7 -tWGGbBi0TRJCa8BkvgngUZxOxUlMfg/7cVxOIB0TPoUSgxFd/+qVz4GZMvr0dPu7 -eAF7J/8ECVvb0wSPTUI1N3c= +MIIG/wIBADANBgkqhkiG9w0BAQEFAASCBukwggblAgEAAoIBgQCylKlLaKU+hOvJ +DfriTRLd+IthG5hv28I3A/CGjLICT0rDDtgaXd0uqloJAnjsgn5gMAcStpDW8Rm+ +t6LsrBL+5fBgkyU1r94Rvx0HHoyaZwBBouitVHw28hP3W+smddkqB1UxpGnTeL2B +gj3dVo/WTtRfO+0h0PKw1l98YE1pMTdqIwcOOE/ER0g4hvA/wrxuLhMvlVLMy/lL +58uctqaDUqryNyeerKbVkq4fJyCG5D2TwXVJ3i2DDh0xSt2Y10poZV4M4k8Su9Z5 +8zN2PSvYMT50aqF277v8BaOeYUApBE4kZGIJpo13ATGdEwpUFZ0Fri4zLYUZ1hWb +OC35sKo7OxWQ/+tefNUdgWHob6Vmy777jiYcLwxc3sS9rF3AJe0rMW83kCkR6hmy +A3250E137N/1QumHuT/Nj9rnI/lwt9jfaYkZjoAgT/C97m/mM83cYpGTdoGV1xNo +7G90MhP0di5FnVsrIaSnvkbGT9UgUWx0oVMjocifdG2qIhMI9psCAwEAAQKCAYBT +sHmaPmNaZj59jZCqp0YVQlpHWwBYQ5vD3pPE6oCttm0p9nXt/VkfenQRTthOtmT1 +POzDp00/feP7zeGLmqSYUjgRekPw4gdnN7Ip2PY5kdW77NWwDSzdLxuOS8Rq1MW9 +/Yu+ZPe3RBlDbT8C0IM+Atlh/BqIQ3zIxN4g0pzUlF0M33d6AYfYSzOcUhibOO7H +j84r+YXBNkIRgYKZYbutRXuZYaGuqejRpBj3voVu0d3Ntdb6lCWuClpB9HzfGN0c +RTv8g6UYO4sK3qyFn90ibIR/1GB9watvtoWVZqggiWeBzSWVWRsGEf9O+Cx4oJw1 +IphglhmhbgNksbj7bD24on/icldSOiVkoUemUOFmHWhCm4PnB1GmbD8YMfEdSbks +qDr1Ps1zg4mGOinVD/4cY7vuPFO/HCH07wfeaUGzRt4g0/yLr+XjVofOA3oowyxv +JAzr+niHA3lg5ecj4r7M68efwzN1OCyjMrVJw2RAzwvGxE+rm5NiT08SWlKQZnkC +gcEA4wvyLpIur/UB84nV3XVJ89UMNBLm++aTFzld047BLJtMaOhvNqx6Cl5c8VuW +l261KHjiVzpfNM3/A2LBQJcYkhX7avkqEXlj57cl+dCWAVwUzKmLJTPjfaTTZnYJ +xeN3dMYjJz2z2WtgvfvDoJLukVwIMmhTY8wtqqYyQBJ/l06pBsfw5TNvmVIOQHds +8ASOiFt+WRLk2bl9xrGGayqt3VV93KVRzF27cpjOgEcG74F3c0ZW9snERN7vIYwB +JfrlAoHBAMlahPwMP2TYylG8OzHe7EiehTekSO26LGh0Cq3wTGXYsK/q8hQCzL14 +kWW638vpwXL6L9ntvrd7hjzWRO3vX/VxnYEA6f0bpqHq1tZi6lzix5CTUN5McpDg +QnjenSJNrNjS1zEF8WeY9iLEuDI/M/iUW4y9R6s3WpgQhPDXpSvd2g3gMGRUYhxQ +Xna8auiJeYFq0oNaOxvJj+VeOfJ3ZMJttd+Y7gTOYZcbg3SdRb/kdxYki0RMD2hF +4ZvjJ6CTfwKBwQDiMqiZFTJGQwYqp4vWEmAW+I4r4xkUpWatoI2Fk5eI5T9+1PLX +uYXsho56NxEU1UrOg4Cb/p+TcBc8PErkGqR0BkpxDMOInTOXSrQe6lxIBoECVXc3 +HTbrmiay0a5y5GfCgxPKqIJhfcToAceoVjovv0y7S4yoxGZKuUEe7E8JY2iqRNAO +yOvKCCICv/hcN235E44RF+2/rDlOltagNej5tY6rIFkaDdgOF4bD7f9O5eEni1Bg +litfoesDtQP/3rECgcEAkQfvQ7D6tIPmbqsbJBfCr6fmoqZllT4FIJN84b50+OL0 +mTGsfjdqC4tdhx3sdu7/VPbaIqm5NmX10bowWgWSY7MbVME4yQPyqSwC5NbIonEC +d6N0mzoLR0kQ+Ai4u+2g82gicgAq2oj1uSNi3WZi48jQjHYFulCbo246o1NgeFFK +77WshYe2R1ioQfQDOU1URKCR0uTaMHClgfu112yiGd12JAD+aF3TM0kxDXz+sXI5 +SKy311DFxECZeXRLpcC3AoHBAJkNMJWTyPYbeVu+CTQkec8Uun233EkXa2kUNZc/ +5DuXDaK+A3DMgYRufTKSPpDHGaCZ1SYPInX1Uoe2dgVjWssRL2uitR4ENabDoAOA +ICVYXYYNagqQu5wwirF0QeaMXo1fjhuuHQh8GsMdXZvYEaAITZ9/NG5x/oY08+8H +kr78SMBOPy3XQn964uKG+e3JwpOG14GKABdAlrHKFXNWchu/6dgcYXB87mrC/GhO +zNwzC+QhFTZoOomFoqMgFWujng== -----END PRIVATE KEY----- diff --git a/Lib/test/test_asyncio/utils.py b/Lib/test/test_asyncio/utils.py index 5362591b5d73..e7438d40a441 100644 --- a/Lib/test/test_asyncio/utils.py +++ b/Lib/test/test_asyncio/utils.py @@ -57,9 +57,9 @@ def data_file(filename): 'issuer': ((('countryName', 'XY'),), (('organizationName', 'Python Software Foundation CA'),), (('commonName', 'our-ca-server'),)), - 'notAfter': 'Nov 28 19:09:06 2027 GMT', - 'notBefore': 'Jan 19 19:09:06 2018 GMT', - 'serialNumber': '82EDBF41C880919C', + 'notAfter': 'Jul 7 14:23:16 2028 GMT', + 'notBefore': 'Aug 29 14:23:16 2018 GMT', + 'serialNumber': 'CB2D80995A69525C', 'subject': ((('countryName', 'XY'),), (('localityName', 'Castle Anthrax'),), (('organizationName', 'Python Software Foundation'),), diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 5c22630d190c..49a4d7295422 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -61,9 +61,9 @@ def data_file(*name): (('localityName', 'Castle Anthrax'),), (('organizationName', 'Python Software Foundation'),), (('commonName', 'localhost'),)), - 'notAfter': 'Jan 17 19:09:06 2028 GMT', - 'notBefore': 'Jan 19 19:09:06 2018 GMT', - 'serialNumber': 'F9BA076D5B6ABD9B', + 'notAfter': 'Aug 26 14:23:15 2028 GMT', + 'notBefore': 'Aug 29 14:23:15 2018 GMT', + 'serialNumber': '98A7CF88C74A32ED', 'subject': ((('countryName', 'XY'),), (('localityName', 'Castle Anthrax'),), (('organizationName', 'Python Software Foundation'),), @@ -86,9 +86,9 @@ def data_file(*name): 'issuer': ((('countryName', 'XY'),), (('organizationName', 'Python Software Foundation CA'),), (('commonName', 'our-ca-server'),)), - 'notAfter': 'Nov 28 19:09:06 2027 GMT', - 'notBefore': 'Jan 19 19:09:06 2018 GMT', - 'serialNumber': '82EDBF41C880919C', + 'notAfter': 'Jul 7 14:23:16 2028 GMT', + 'notBefore': 'Aug 29 14:23:16 2018 GMT', + 'serialNumber': 'CB2D80995A69525C', 'subject': ((('countryName', 'XY'),), (('localityName', 'Castle Anthrax'),), (('organizationName', 'Python Software Foundation'),), @@ -334,6 +334,8 @@ def test_random_fork(self): self.assertNotEqual(child_random, parent_random) + maxDiff = None + def test_parse_cert(self): # note that this uses an 'unofficial' function in _ssl.c, # provided solely for this test, to exercise the certificate diff --git a/Misc/NEWS.d/next/Tests/2018-08-29-16-30-52.bpo-34542.9stVAW.rst b/Misc/NEWS.d/next/Tests/2018-08-29-16-30-52.bpo-34542.9stVAW.rst new file mode 100644 index 000000000000..1ca3c7d7996c --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2018-08-29-16-30-52.bpo-34542.9stVAW.rst @@ -0,0 +1 @@ +Use 3072 RSA keys and SHA-256 signature for test certs and keys. From webhook-mailer at python.org Thu Sep 6 14:43:41 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Thu, 06 Sep 2018 18:43:41 -0000 Subject: [Python-checkins] closes bpo-34594: Don't hardcode errno values in the tests. (GH-9076) Message-ID: https://github.com/python/cpython/commit/b03c2c51909e3b5b5966d86a2829b5ddf2d496aa commit: b03c2c51909e3b5b5966d86a2829b5ddf2d496aa branch: master author: Zackery Spytz committer: Benjamin Peterson date: 2018-09-06T11:43:30-07:00 summary: closes bpo-34594: Don't hardcode errno values in the tests. (GH-9076) files: A Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst M Lib/test/test_spwd.py M Lib/test/test_tabnanny.py diff --git a/Lib/test/test_spwd.py b/Lib/test/test_spwd.py index e893f3a847fd..07793c84c8e9 100644 --- a/Lib/test/test_spwd.py +++ b/Lib/test/test_spwd.py @@ -67,8 +67,6 @@ def test_getspnam_exception(self): spwd.getspnam(name) except KeyError as exc: self.skipTest("spwd entry %r doesn't exist: %s" % (name, exc)) - else: - self.assertEqual(str(cm.exception), '[Errno 13] Permission denied') if __name__ == "__main__": diff --git a/Lib/test/test_tabnanny.py b/Lib/test/test_tabnanny.py index ec887361730b..845096e63c26 100644 --- a/Lib/test/test_tabnanny.py +++ b/Lib/test/test_tabnanny.py @@ -5,6 +5,7 @@ """ from unittest import TestCase, mock from unittest import mock +import errno import tabnanny import tokenize import tempfile @@ -232,7 +233,8 @@ def test_when_nannynag_error(self): def test_when_no_file(self): """A python file which does not exist actually in system.""" path = 'no_file.py' - err = f"{path!r}: I/O Error: [Errno 2] No such file or directory: {path!r}\n" + err = f"{path!r}: I/O Error: [Errno {errno.ENOENT}] " \ + f"No such file or directory: {path!r}\n" self.verify_tabnanny_check(path, err=err) def test_errored_directory(self): diff --git a/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst b/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst new file mode 100644 index 000000000000..7a7b1f055561 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst @@ -0,0 +1 @@ +Fix usage of hardcoded ``errno`` values in the tests. From webhook-mailer at python.org Fri Sep 7 00:54:53 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Fri, 07 Sep 2018 04:54:53 -0000 Subject: [Python-checkins] closes bpo-34599: Improve performance of _Py_bytes_capitalize(). (GH-9083) Message-ID: https://github.com/python/cpython/commit/593bb30e82eded7f2ec02f7d1aa49742e6962113 commit: 593bb30e82eded7f2ec02f7d1aa49742e6962113 branch: master author: Sergey Fedoseev committer: Benjamin Peterson date: 2018-09-06T21:54:49-07:00 summary: closes bpo-34599: Improve performance of _Py_bytes_capitalize(). (GH-9083) files: M Objects/bytes_methods.c diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c index 05679e31c9d6..37c5f7dbc804 100644 --- a/Objects/bytes_methods.c +++ b/Objects/bytes_methods.c @@ -361,23 +361,9 @@ and the rest lower-cased."); void _Py_bytes_capitalize(char *result, const char *s, Py_ssize_t len) { - Py_ssize_t i; - - if (0 < len) { - int c = Py_CHARMASK(*s++); - if (Py_ISLOWER(c)) - *result = Py_TOUPPER(c); - else - *result = c; - result++; - } - for (i = 1; i < len; i++) { - int c = Py_CHARMASK(*s++); - if (Py_ISUPPER(c)) - *result = Py_TOLOWER(c); - else - *result = c; - result++; + if (len > 0) { + *result = Py_TOUPPER(*s); + _Py_bytes_lower(result + 1, s + 1, len - 1); } } From webhook-mailer at python.org Fri Sep 7 03:37:05 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 07:37:05 -0000 Subject: [Python-checkins] bpo-25750: fix refcounts in type_getattro() (GH-6118) Message-ID: https://github.com/python/cpython/commit/8f735485acf2e35a75d2fa019feb8f905598c4e5 commit: 8f735485acf2e35a75d2fa019feb8f905598c4e5 branch: master author: jdemeyer committer: Victor Stinner date: 2018-09-07T09:37:00+02:00 summary: bpo-25750: fix refcounts in type_getattro() (GH-6118) When calling tp_descr_get(self, obj, type), make sure that we own a strong reference to "self". files: A Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst M Objects/typeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst new file mode 100644 index 000000000000..09ffb368b7c6 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst @@ -0,0 +1,2 @@ +Fix rare Python crash due to bad refcounting in ``type_getattro()`` if a +descriptor deletes itself from the class. Patch by Jeroen Demeyer. diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 52fcfeb22871..b9e69bf1bd1d 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3145,6 +3145,7 @@ type_getattro(PyTypeObject *type, PyObject *name) PyTypeObject *metatype = Py_TYPE(type); PyObject *meta_attribute, *attribute; descrgetfunc meta_get; + PyObject* res; if (!PyUnicode_Check(name)) { PyErr_Format(PyExc_TypeError, @@ -3166,6 +3167,7 @@ type_getattro(PyTypeObject *type, PyObject *name) meta_attribute = _PyType_Lookup(metatype, name); if (meta_attribute != NULL) { + Py_INCREF(meta_attribute); meta_get = Py_TYPE(meta_attribute)->tp_descr_get; if (meta_get != NULL && PyDescr_IsData(meta_attribute)) { @@ -3173,10 +3175,11 @@ type_getattro(PyTypeObject *type, PyObject *name) * writes. Assume the attribute is not overridden in * type's tp_dict (and bases): call the descriptor now. */ - return meta_get(meta_attribute, (PyObject *)type, - (PyObject *)metatype); + res = meta_get(meta_attribute, (PyObject *)type, + (PyObject *)metatype); + Py_DECREF(meta_attribute); + return res; } - Py_INCREF(meta_attribute); } /* No data descriptor found on metatype. Look in tp_dict of this @@ -3184,6 +3187,7 @@ type_getattro(PyTypeObject *type, PyObject *name) attribute = _PyType_Lookup(type, name); if (attribute != NULL) { /* Implement descriptor functionality, if any */ + Py_INCREF(attribute); descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get; Py_XDECREF(meta_attribute); @@ -3191,11 +3195,12 @@ type_getattro(PyTypeObject *type, PyObject *name) if (local_get != NULL) { /* NULL 2nd argument indicates the descriptor was * found on the target object itself (or a base) */ - return local_get(attribute, (PyObject *)NULL, - (PyObject *)type); + res = local_get(attribute, (PyObject *)NULL, + (PyObject *)type); + Py_DECREF(attribute); + return res; } - Py_INCREF(attribute); return attribute; } From webhook-mailer at python.org Fri Sep 7 03:50:44 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 07 Sep 2018 07:50:44 -0000 Subject: [Python-checkins] bpo-25750: fix refcounts in type_getattro() (GH-6118) Message-ID: https://github.com/python/cpython/commit/f862f3abaed59b83763707ae529f0fe487961ba9 commit: f862f3abaed59b83763707ae529f0fe487961ba9 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-07T00:50:35-07:00 summary: bpo-25750: fix refcounts in type_getattro() (GH-6118) When calling tp_descr_get(self, obj, type), make sure that we own a strong reference to "self". (cherry picked from commit 8f735485acf2e35a75d2fa019feb8f905598c4e5) Co-authored-by: jdemeyer files: A Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst M Objects/typeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst new file mode 100644 index 000000000000..09ffb368b7c6 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst @@ -0,0 +1,2 @@ +Fix rare Python crash due to bad refcounting in ``type_getattro()`` if a +descriptor deletes itself from the class. Patch by Jeroen Demeyer. diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 52fcfeb22871..b9e69bf1bd1d 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3145,6 +3145,7 @@ type_getattro(PyTypeObject *type, PyObject *name) PyTypeObject *metatype = Py_TYPE(type); PyObject *meta_attribute, *attribute; descrgetfunc meta_get; + PyObject* res; if (!PyUnicode_Check(name)) { PyErr_Format(PyExc_TypeError, @@ -3166,6 +3167,7 @@ type_getattro(PyTypeObject *type, PyObject *name) meta_attribute = _PyType_Lookup(metatype, name); if (meta_attribute != NULL) { + Py_INCREF(meta_attribute); meta_get = Py_TYPE(meta_attribute)->tp_descr_get; if (meta_get != NULL && PyDescr_IsData(meta_attribute)) { @@ -3173,10 +3175,11 @@ type_getattro(PyTypeObject *type, PyObject *name) * writes. Assume the attribute is not overridden in * type's tp_dict (and bases): call the descriptor now. */ - return meta_get(meta_attribute, (PyObject *)type, - (PyObject *)metatype); + res = meta_get(meta_attribute, (PyObject *)type, + (PyObject *)metatype); + Py_DECREF(meta_attribute); + return res; } - Py_INCREF(meta_attribute); } /* No data descriptor found on metatype. Look in tp_dict of this @@ -3184,6 +3187,7 @@ type_getattro(PyTypeObject *type, PyObject *name) attribute = _PyType_Lookup(type, name); if (attribute != NULL) { /* Implement descriptor functionality, if any */ + Py_INCREF(attribute); descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get; Py_XDECREF(meta_attribute); @@ -3191,11 +3195,12 @@ type_getattro(PyTypeObject *type, PyObject *name) if (local_get != NULL) { /* NULL 2nd argument indicates the descriptor was * found on the target object itself (or a base) */ - return local_get(attribute, (PyObject *)NULL, - (PyObject *)type); + res = local_get(attribute, (PyObject *)NULL, + (PyObject *)type); + Py_DECREF(attribute); + return res; } - Py_INCREF(attribute); return attribute; } From webhook-mailer at python.org Fri Sep 7 03:57:48 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 07:57:48 -0000 Subject: [Python-checkins] bpo-25750: fix refcounts in type_getattro() (GH-6118) (GH-9088) Message-ID: https://github.com/python/cpython/commit/3ee07432f2e607cc6e7e6ea2d3695b672ceb1cea commit: 3ee07432f2e607cc6e7e6ea2d3695b672ceb1cea branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-07T09:57:45+02:00 summary: bpo-25750: fix refcounts in type_getattro() (GH-6118) (GH-9088) When calling tp_descr_get(self, obj, type), make sure that we own a strong reference to "self". (cherry picked from commit 8f735485acf2e35a75d2fa019feb8f905598c4e5) Co-authored-by: jdemeyer files: A Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst M Objects/typeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst new file mode 100644 index 000000000000..09ffb368b7c6 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst @@ -0,0 +1,2 @@ +Fix rare Python crash due to bad refcounting in ``type_getattro()`` if a +descriptor deletes itself from the class. Patch by Jeroen Demeyer. diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 3468c64060df..ed2d40064adc 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3004,6 +3004,7 @@ type_getattro(PyTypeObject *type, PyObject *name) PyTypeObject *metatype = Py_TYPE(type); PyObject *meta_attribute, *attribute; descrgetfunc meta_get; + PyObject* res; if (!PyUnicode_Check(name)) { PyErr_Format(PyExc_TypeError, @@ -3025,6 +3026,7 @@ type_getattro(PyTypeObject *type, PyObject *name) meta_attribute = _PyType_Lookup(metatype, name); if (meta_attribute != NULL) { + Py_INCREF(meta_attribute); meta_get = Py_TYPE(meta_attribute)->tp_descr_get; if (meta_get != NULL && PyDescr_IsData(meta_attribute)) { @@ -3032,10 +3034,11 @@ type_getattro(PyTypeObject *type, PyObject *name) * writes. Assume the attribute is not overridden in * type's tp_dict (and bases): call the descriptor now. */ - return meta_get(meta_attribute, (PyObject *)type, - (PyObject *)metatype); + res = meta_get(meta_attribute, (PyObject *)type, + (PyObject *)metatype); + Py_DECREF(meta_attribute); + return res; } - Py_INCREF(meta_attribute); } /* No data descriptor found on metatype. Look in tp_dict of this @@ -3043,6 +3046,7 @@ type_getattro(PyTypeObject *type, PyObject *name) attribute = _PyType_Lookup(type, name); if (attribute != NULL) { /* Implement descriptor functionality, if any */ + Py_INCREF(attribute); descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get; Py_XDECREF(meta_attribute); @@ -3050,11 +3054,12 @@ type_getattro(PyTypeObject *type, PyObject *name) if (local_get != NULL) { /* NULL 2nd argument indicates the descriptor was * found on the target object itself (or a base) */ - return local_get(attribute, (PyObject *)NULL, - (PyObject *)type); + res = local_get(attribute, (PyObject *)NULL, + (PyObject *)type); + Py_DECREF(attribute); + return res; } - Py_INCREF(attribute); return attribute; } From webhook-mailer at python.org Fri Sep 7 04:15:42 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 08:15:42 -0000 Subject: [Python-checkins] bpo-25750: fix refcounts in type_getattro() (GH-6118) (GH-9091) Message-ID: https://github.com/python/cpython/commit/bf2bd8f8a1d88de60c114de957f50fe2433e3937 commit: bf2bd8f8a1d88de60c114de957f50fe2433e3937 branch: 2.7 author: Victor Stinner committer: GitHub date: 2018-09-07T10:15:31+02:00 summary: bpo-25750: fix refcounts in type_getattro() (GH-6118) (GH-9091) When calling tp_descr_get(self, obj, type), make sure that we own a strong reference to "self". (cherry picked from commit 8f735485acf2e35a75d2fa019feb8f905598c4e5) files: A Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst M Objects/typeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst new file mode 100644 index 000000000000..09ffb368b7c6 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-03-14-21-42-17.bpo-25750.lxgkQz.rst @@ -0,0 +1,2 @@ +Fix rare Python crash due to bad refcounting in ``type_getattro()`` if a +descriptor deletes itself from the class. Patch by Jeroen Demeyer. diff --git a/Objects/typeobject.c b/Objects/typeobject.c index d576b8250d01..844fb0074920 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2611,6 +2611,7 @@ type_getattro(PyTypeObject *type, PyObject *name) PyTypeObject *metatype = Py_TYPE(type); PyObject *meta_attribute, *attribute; descrgetfunc meta_get; + PyObject* res; if (!PyString_Check(name)) { PyErr_Format(PyExc_TypeError, @@ -2632,6 +2633,7 @@ type_getattro(PyTypeObject *type, PyObject *name) meta_attribute = _PyType_Lookup(metatype, name); if (meta_attribute != NULL) { + Py_INCREF(meta_attribute); meta_get = Py_TYPE(meta_attribute)->tp_descr_get; if (meta_get != NULL && PyDescr_IsData(meta_attribute)) { @@ -2639,10 +2641,11 @@ type_getattro(PyTypeObject *type, PyObject *name) * writes. Assume the attribute is not overridden in * type's tp_dict (and bases): call the descriptor now. */ - return meta_get(meta_attribute, (PyObject *)type, - (PyObject *)metatype); + res = meta_get(meta_attribute, (PyObject *)type, + (PyObject *)metatype); + Py_DECREF(meta_attribute); + return res; } - Py_INCREF(meta_attribute); } /* No data descriptor found on metatype. Look in tp_dict of this @@ -2650,18 +2653,21 @@ type_getattro(PyTypeObject *type, PyObject *name) attribute = _PyType_Lookup(type, name); if (attribute != NULL) { /* Implement descriptor functionality, if any */ - descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get; + descrgetfunc local_get; + Py_INCREF(attribute); + local_get = Py_TYPE(attribute)->tp_descr_get; Py_XDECREF(meta_attribute); if (local_get != NULL) { /* NULL 2nd argument indicates the descriptor was * found on the target object itself (or a base) */ - return local_get(attribute, (PyObject *)NULL, - (PyObject *)type); + res = local_get(attribute, (PyObject *)NULL, + (PyObject *)type); + Py_DECREF(attribute); + return res; } - Py_INCREF(attribute); return attribute; } From webhook-mailer at python.org Fri Sep 7 04:56:17 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 08:56:17 -0000 Subject: [Python-checkins] _sre.c: Removed unused SRE_IS_ALNUM macro (GH-9090) Message-ID: https://github.com/python/cpython/commit/f9925d86c9d7a44b612576131cd5b84cfee36c4e commit: f9925d86c9d7a44b612576131cd5b84cfee36c4e branch: master author: Sergey Fedoseev committer: Victor Stinner date: 2018-09-07T10:56:09+02:00 summary: _sre.c: Removed unused SRE_IS_ALNUM macro (GH-9090) files: M Modules/_sre.c diff --git a/Modules/_sre.c b/Modules/_sre.c index b6be6f6ffa60..d67083037e51 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -92,8 +92,6 @@ static const char copyright[] = ((ch) < 128 && Py_ISSPACE(ch)) #define SRE_IS_LINEBREAK(ch)\ ((ch) == '\n') -#define SRE_IS_ALNUM(ch)\ - ((ch) < 128 && Py_ISALNUM(ch)) #define SRE_IS_WORD(ch)\ ((ch) < 128 && (Py_ISALNUM(ch) || (ch) == '_')) From solipsis at pitrou.net Fri Sep 7 05:08:53 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 07 Sep 2018 09:08:53 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=3 Message-ID: <20180907090853.1.E9F414B1C474F2B0@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [7, -7, 1] memory blocks, sum=1 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_fork leaked [0, -2, 0] memory blocks, sum=-2 test_multiprocessing_spawn leaked [-2, 1, 1] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflog1l3pN0', '--timeout', '7200'] From webhook-mailer at python.org Fri Sep 7 05:31:52 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 09:31:52 -0000 Subject: [Python-checkins] Doc: Missing 'f' in an f-string. (GH-9074) Message-ID: https://github.com/python/cpython/commit/25fa141487e61b94f15289619cb3af764cf65e58 commit: 25fa141487e61b94f15289619cb3af764cf65e58 branch: master author: Julien Palard committer: Victor Stinner date: 2018-09-07T11:31:47+02:00 summary: Doc: Missing 'f' in an f-string. (GH-9074) files: M Doc/tutorial/inputoutput.rst diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index dfe4c3849cbe..a92c26681596 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -127,7 +127,7 @@ applies :func:`repr`:: >>> animals = 'eels' >>> print(f'My hovercraft is full of {animals}.') My hovercraft is full of eels. - >>> print('My hovercraft is full of {animals !r}.') + >>> print(f'My hovercraft is full of {animals !r}.') My hovercraft is full of 'eels'. For a reference on these format specifications, see From webhook-mailer at python.org Fri Sep 7 05:59:05 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 09:59:05 -0000 Subject: [Python-checkins] Doc: Missing 'f' in an f-string. (GH-9074) (GH-9095) Message-ID: https://github.com/python/cpython/commit/854b740910f20441b6a42c2047d5f0ad0451404e commit: 854b740910f20441b6a42c2047d5f0ad0451404e branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Victor Stinner date: 2018-09-07T11:59:00+02:00 summary: Doc: Missing 'f' in an f-string. (GH-9074) (GH-9095) (cherry picked from commit 25fa141487e61b94f15289619cb3af764cf65e58) Co-authored-by: Julien Palard files: M Doc/tutorial/inputoutput.rst diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index dfe4c3849cbe..a92c26681596 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -127,7 +127,7 @@ applies :func:`repr`:: >>> animals = 'eels' >>> print(f'My hovercraft is full of {animals}.') My hovercraft is full of eels. - >>> print('My hovercraft is full of {animals !r}.') + >>> print(f'My hovercraft is full of {animals !r}.') My hovercraft is full of 'eels'. For a reference on these format specifications, see From webhook-mailer at python.org Fri Sep 7 08:06:24 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 12:06:24 -0000 Subject: [Python-checkins] bpo-33625: Release GIL for grp.getgr{nam, gid} and pwd.getpw{nam, uid} (GH-7081) Message-ID: https://github.com/python/cpython/commit/23e65b25557f957af840cf8fe68e80659ce28629 commit: 23e65b25557f957af840cf8fe68e80659ce28629 branch: master author: William Grzybowski committer: Victor Stinner date: 2018-09-07T14:06:15+02:00 summary: bpo-33625: Release GIL for grp.getgr{nam,gid} and pwd.getpw{nam,uid} (GH-7081) Release GIL on grp.getgrnam(), grp.getgrgid(), pwd.getpwnam() and pwd.getpwuid() if reentrant variants of these functions are available. Patch by William Grzybowski. files: A Misc/NEWS.d/next/Library/2018-05-23-17-07-54.bpo-33625.nzQgD8.rst M Modules/grpmodule.c M Modules/pwdmodule.c M configure M configure.ac M pyconfig.h.in diff --git a/Misc/NEWS.d/next/Library/2018-05-23-17-07-54.bpo-33625.nzQgD8.rst b/Misc/NEWS.d/next/Library/2018-05-23-17-07-54.bpo-33625.nzQgD8.rst new file mode 100644 index 000000000000..265fab2289c2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-05-23-17-07-54.bpo-33625.nzQgD8.rst @@ -0,0 +1,3 @@ +Release GIL on `grp.getgrnam`, `grp.getgrgid`, `pwd.getpwnam` and +`pwd.getpwuid` if reentrant variants of these functions are available. +Patch by William Grzybowski. diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c index f577fd3ab4ec..ffdfa1b6f9fb 100644 --- a/Modules/grpmodule.c +++ b/Modules/grpmodule.c @@ -37,6 +37,8 @@ static PyStructSequence_Desc struct_group_type_desc = { static int initialized; static PyTypeObject StructGrpType; +#define DEFAULT_BUFFER_SIZE 1024 + static PyObject * mkgrent(struct group *p) { @@ -96,7 +98,9 @@ static PyObject * grp_getgrgid_impl(PyObject *module, PyObject *id) /*[clinic end generated code: output=30797c289504a1ba input=15fa0e2ccf5cda25]*/ { - PyObject *py_int_id; + PyObject *py_int_id, *retval = NULL; + int nomem = 0; + char *buf = NULL, *buf2 = NULL; gid_t gid; struct group *p; @@ -119,8 +123,48 @@ grp_getgrgid_impl(PyObject *module, PyObject *id) } Py_DECREF(py_int_id); } +#ifdef HAVE_GETGRGID_R + Py_BEGIN_ALLOW_THREADS + int status; + Py_ssize_t bufsize; + struct group grp; + + bufsize = sysconf(_SC_GETGR_R_SIZE_MAX); + if (bufsize == -1) { + bufsize = DEFAULT_BUFFER_SIZE; + } + + while (1) { + buf2 = PyMem_RawRealloc(buf, bufsize); + if (buf2 == NULL) { + p = NULL; + nomem = 1; + break; + } + buf = buf2; + status = getgrgid_r(gid, &grp, buf, bufsize, &p); + if (status != 0) { + p = NULL; + } + if (p != NULL || status != ERANGE) { + break; + } + if (bufsize > (PY_SSIZE_T_MAX >> 1)) { + nomem = 1; + break; + } + bufsize <<= 1; + } - if ((p = getgrgid(gid)) == NULL) { + Py_END_ALLOW_THREADS +#else + p = getgrgid(gid); +#endif + if (p == NULL) { + PyMem_RawFree(buf); + if (nomem == 1) { + return PyErr_NoMemory(); + } PyObject *gid_obj = _PyLong_FromGid(gid); if (gid_obj == NULL) return NULL; @@ -128,7 +172,11 @@ grp_getgrgid_impl(PyObject *module, PyObject *id) Py_DECREF(gid_obj); return NULL; } - return mkgrent(p); + retval = mkgrent(p); +#ifdef HAVE_GETGRGID_R + PyMem_RawFree(buf); +#endif + return retval; } /*[clinic input] @@ -145,7 +193,8 @@ static PyObject * grp_getgrnam_impl(PyObject *module, PyObject *name) /*[clinic end generated code: output=67905086f403c21c input=08ded29affa3c863]*/ { - char *name_chars; + char *buf = NULL, *buf2 = NULL, *name_chars; + int nomem = 0; struct group *p; PyObject *bytes, *retval = NULL; @@ -154,13 +203,55 @@ grp_getgrnam_impl(PyObject *module, PyObject *name) /* check for embedded null bytes */ if (PyBytes_AsStringAndSize(bytes, &name_chars, NULL) == -1) goto out; +#ifdef HAVE_GETGRNAM_R + Py_BEGIN_ALLOW_THREADS + int status; + Py_ssize_t bufsize; + struct group grp; + + bufsize = sysconf(_SC_GETGR_R_SIZE_MAX); + if (bufsize == -1) { + bufsize = DEFAULT_BUFFER_SIZE; + } + + while(1) { + buf2 = PyMem_RawRealloc(buf, bufsize); + if (buf2 == NULL) { + p = NULL; + nomem = 1; + break; + } + buf = buf2; + status = getgrnam_r(name_chars, &grp, buf, bufsize, &p); + if (status != 0) { + p = NULL; + } + if (p != NULL || status != ERANGE) { + break; + } + if (bufsize > (PY_SSIZE_T_MAX >> 1)) { + nomem = 1; + break; + } + bufsize <<= 1; + } - if ((p = getgrnam(name_chars)) == NULL) { - PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %s", name_chars); + Py_END_ALLOW_THREADS +#else + p = getgrnam(name_chars); +#endif + if (p == NULL) { + if (nomem == 1) { + PyErr_NoMemory(); + } + else { + PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %s", name_chars); + } goto out; } retval = mkgrent(p); out: + PyMem_RawFree(buf); Py_DECREF(bytes); return retval; } diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c index bbef2de9c522..36192a570433 100644 --- a/Modules/pwdmodule.c +++ b/Modules/pwdmodule.c @@ -50,6 +50,8 @@ exception is raised if the entry asked for cannot be found."); static int initialized; static PyTypeObject StructPwdType; +#define DEFAULT_BUFFER_SIZE 1024 + static void sets(PyObject *v, int i, const char* val) { @@ -116,8 +118,11 @@ static PyObject * pwd_getpwuid(PyObject *module, PyObject *uidobj) /*[clinic end generated code: output=c4ee1d4d429b86c4 input=ae64d507a1c6d3e8]*/ { + PyObject *retval = NULL; uid_t uid; + int nomem = 0; struct passwd *p; + char *buf = NULL, *buf2 = NULL; if (!_Py_Uid_Converter(uidobj, &uid)) { if (PyErr_ExceptionMatches(PyExc_OverflowError)) @@ -125,7 +130,47 @@ pwd_getpwuid(PyObject *module, PyObject *uidobj) "getpwuid(): uid not found"); return NULL; } - if ((p = getpwuid(uid)) == NULL) { +#ifdef HAVE_GETPWUID_R + Py_BEGIN_ALLOW_THREADS + int status; + Py_ssize_t bufsize; + struct passwd pwd; + + bufsize = sysconf(_SC_GETPW_R_SIZE_MAX); + if (bufsize == -1) { + bufsize = DEFAULT_BUFFER_SIZE; + } + + while(1) { + buf2 = PyMem_RawRealloc(buf, bufsize); + if (buf2 == NULL) { + nomem = 1; + break; + } + buf = buf2; + status = getpwuid_r(uid, &pwd, buf, bufsize, &p); + if (status != 0) { + p = NULL; + } + if (p != NULL || status != ERANGE) { + break; + } + if (bufsize > (PY_SSIZE_T_MAX >> 1)) { + nomem = 1; + break; + } + bufsize <<= 1; + } + + Py_END_ALLOW_THREADS +#else + p = getpwuid(uid); +#endif + if (p == NULL) { + PyMem_RawFree(buf); + if (nomem == 1) { + return PyErr_NoMemory(); + } PyObject *uid_obj = _PyLong_FromUid(uid); if (uid_obj == NULL) return NULL; @@ -134,7 +179,11 @@ pwd_getpwuid(PyObject *module, PyObject *uidobj) Py_DECREF(uid_obj); return NULL; } - return mkpwent(p); + retval = mkpwent(p); +#ifdef HAVE_GETPWUID_R + PyMem_RawFree(buf); +#endif + return retval; } /*[clinic input] @@ -152,7 +201,8 @@ static PyObject * pwd_getpwnam_impl(PyObject *module, PyObject *arg) /*[clinic end generated code: output=6abeee92430e43d2 input=d5f7e700919b02d3]*/ { - char *name; + char *buf = NULL, *buf2 = NULL, *name; + int nomem = 0; struct passwd *p; PyObject *bytes, *retval = NULL; @@ -161,13 +211,55 @@ pwd_getpwnam_impl(PyObject *module, PyObject *arg) /* check for embedded null bytes */ if (PyBytes_AsStringAndSize(bytes, &name, NULL) == -1) goto out; - if ((p = getpwnam(name)) == NULL) { - PyErr_Format(PyExc_KeyError, - "getpwnam(): name not found: %s", name); +#ifdef HAVE_GETPWNAM_R + Py_BEGIN_ALLOW_THREADS + int status; + Py_ssize_t bufsize; + struct passwd pwd; + + bufsize = sysconf(_SC_GETPW_R_SIZE_MAX); + if (bufsize == -1) { + bufsize = DEFAULT_BUFFER_SIZE; + } + + while(1) { + buf2 = PyMem_RawRealloc(buf, bufsize); + if (buf2 == NULL) { + nomem = 1; + break; + } + buf = buf2; + status = getpwnam_r(name, &pwd, buf, bufsize, &p); + if (status != 0) { + p = NULL; + } + if (p != NULL || status != ERANGE) { + break; + } + if (bufsize > (PY_SSIZE_T_MAX >> 1)) { + nomem = 1; + break; + } + bufsize <<= 1; + } + + Py_END_ALLOW_THREADS +#else + p = getpwnam(name); +#endif + if (p == NULL) { + if (nomem == 1) { + PyErr_NoMemory(); + } + else { + PyErr_Format(PyExc_KeyError, + "getpwnam(): name not found: %s", name); + } goto out; } retval = mkpwent(p); out: + PyMem_RawFree(buf); Py_DECREF(bytes); return retval; } diff --git a/configure b/configure index 0b07f0b97bf1..b4a70e6736bb 100755 --- a/configure +++ b/configure @@ -781,6 +781,7 @@ infodir docdir oldincludedir includedir +runstatedir localstatedir sharedstatedir sysconfdir @@ -890,6 +891,7 @@ datadir='${datarootdir}' sysconfdir='${prefix}/etc' sharedstatedir='${prefix}/com' localstatedir='${prefix}/var' +runstatedir='${localstatedir}/run' includedir='${prefix}/include' oldincludedir='/usr/include' docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' @@ -1142,6 +1144,15 @@ do | -silent | --silent | --silen | --sile | --sil) silent=yes ;; + -runstatedir | --runstatedir | --runstatedi | --runstated \ + | --runstate | --runstat | --runsta | --runst | --runs \ + | --run | --ru | --r) + ac_prev=runstatedir ;; + -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \ + | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \ + | --run=* | --ru=* | --r=*) + runstatedir=$ac_optarg ;; + -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb) ac_prev=sbindir ;; -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \ @@ -1279,7 +1290,7 @@ fi for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \ datadir sysconfdir sharedstatedir localstatedir includedir \ oldincludedir docdir infodir htmldir dvidir pdfdir psdir \ - libdir localedir mandir + libdir localedir mandir runstatedir do eval ac_val=\$$ac_var # Remove trailing slashes. @@ -1432,6 +1443,7 @@ Fine tuning of the installation directories: --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] + --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] @@ -11242,8 +11254,9 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ clock confstr ctermid dup3 execv faccessat fchmod fchmodat fchown fchownat \ fexecve fdopendir fork fpathconf fstatat ftime ftruncate futimesat \ futimens futimes gai_strerror getentropy \ + getgrgid_r getgrnam_r \ getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \ - getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \ + getpriority getresuid getresgid getpwent getpwnam_r getpwuid_r getspnam getspent getsid getwd \ if_nameindex \ initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \ memrchr mbrtowc mkdirat mkfifo \ @@ -13675,6 +13688,7 @@ fi + # checks for system services # (none yet) diff --git a/configure.ac b/configure.ac index 5b66624d7819..e47586bd403c 100644 --- a/configure.ac +++ b/configure.ac @@ -3434,8 +3434,9 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ clock confstr ctermid dup3 execv faccessat fchmod fchmodat fchown fchownat \ fexecve fdopendir fork fpathconf fstatat ftime ftruncate futimesat \ futimens futimes gai_strerror getentropy \ + getgrgid_r getgrnam_r \ getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \ - getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \ + getpriority getresuid getresgid getpwent getpwnam_r getpwuid_r getspnam getspent getsid getwd \ if_nameindex \ initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \ memrchr mbrtowc mkdirat mkfifo \ diff --git a/pyconfig.h.in b/pyconfig.h.in index a82c3737c3c4..4d3a4b995fe3 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -403,6 +403,12 @@ /* Define to 1 if you have the `getentropy' function. */ #undef HAVE_GETENTROPY +/* Define to 1 if you have the `getgrgid_r' function. */ +#undef HAVE_GETGRGID_R + +/* Define to 1 if you have the `getgrnam_r' function. */ +#undef HAVE_GETGRNAM_R + /* Define to 1 if you have the `getgrouplist' function. */ #undef HAVE_GETGROUPLIST @@ -457,6 +463,12 @@ /* Define to 1 if you have the `getpwent' function. */ #undef HAVE_GETPWENT +/* Define to 1 if you have the `getpwnam_r' function. */ +#undef HAVE_GETPWNAM_R + +/* Define to 1 if you have the `getpwuid_r' function. */ +#undef HAVE_GETPWUID_R + /* Define to 1 if the getrandom() function is available */ #undef HAVE_GETRANDOM From webhook-mailer at python.org Fri Sep 7 08:17:24 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 12:17:24 -0000 Subject: [Python-checkins] [3.6] bpo-34563: Fix for invalid assert on big output of multiprocessing.Process (GH-9027) (GH-9069) Message-ID: https://github.com/python/cpython/commit/23f427a0fdb888212136cf8745a9f5f832a3f374 commit: 23f427a0fdb888212136cf8745a9f5f832a3f374 branch: 3.6 author: Alexander Buchkovsky committer: Victor Stinner date: 2018-09-07T14:17:18+02:00 summary: [3.6] bpo-34563: Fix for invalid assert on big output of multiprocessing.Process (GH-9027) (GH-9069) Fix for invalid assert on big output of multiprocessing.Process. (cherry picked from commit 266f4904a222a784080e29aad0916849e507515d) files: A Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst M Modules/_winapi.c M Modules/clinic/_winapi.c.h diff --git a/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst b/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst new file mode 100644 index 000000000000..9127af0d1924 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-01-20-43-10.bpo-34563.7NQK7B.rst @@ -0,0 +1 @@ +On Windows, fix multiprocessing.Connection for very large read: fix _winapi.PeekNamedPipe() and _winapi.ReadFile() for read larger than INT_MAX (usually 2^31-1). \ No newline at end of file diff --git a/Modules/_winapi.c b/Modules/_winapi.c index b98e7789b787..0647f6bedf1b 100644 --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -1167,7 +1167,7 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) } if (_PyBytes_Resize(&buf, nread)) return NULL; - return Py_BuildValue("Nii", buf, navail, nleft); + return Py_BuildValue("NII", buf, navail, nleft); } else { Py_BEGIN_ALLOW_THREADS @@ -1176,7 +1176,7 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) if (!ret) { return PyErr_SetExcFromWindowsErr(PyExc_IOError, 0); } - return Py_BuildValue("ii", navail, nleft); + return Py_BuildValue("II", navail, nleft); } } @@ -1184,14 +1184,14 @@ _winapi_PeekNamedPipe_impl(PyObject *module, HANDLE handle, int size) _winapi.ReadFile handle: HANDLE - size: int + size: DWORD overlapped as use_overlapped: int(c_default='0') = False [clinic start generated code]*/ static PyObject * -_winapi_ReadFile_impl(PyObject *module, HANDLE handle, int size, +_winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, int use_overlapped) -/*[clinic end generated code: output=492029ca98161d84 input=8dd810194e86ac7d]*/ +/*[clinic end generated code: output=d3d5b44a8201b944 input=1b7d0ed0de1e50bc]*/ { DWORD nread; PyObject *buf; diff --git a/Modules/clinic/_winapi.c.h b/Modules/clinic/_winapi.c.h index 5bfbaf0d5634..feb98bc797d1 100644 --- a/Modules/clinic/_winapi.c.h +++ b/Modules/clinic/_winapi.c.h @@ -673,7 +673,7 @@ PyDoc_STRVAR(_winapi_ReadFile__doc__, {"ReadFile", (PyCFunction)_winapi_ReadFile, METH_FASTCALL, _winapi_ReadFile__doc__}, static PyObject * -_winapi_ReadFile_impl(PyObject *module, HANDLE handle, int size, +_winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, int use_overlapped); static PyObject * @@ -681,9 +681,9 @@ _winapi_ReadFile(PyObject *module, PyObject **args, Py_ssize_t nargs, PyObject * { PyObject *return_value = NULL; static const char * const _keywords[] = {"handle", "size", "overlapped", NULL}; - static _PyArg_Parser _parser = {"" F_HANDLE "i|i:ReadFile", _keywords, 0}; + static _PyArg_Parser _parser = {"" F_HANDLE "k|i:ReadFile", _keywords, 0}; HANDLE handle; - int size; + DWORD size; int use_overlapped = 0; if (!_PyArg_ParseStack(args, nargs, kwnames, &_parser, @@ -889,4 +889,4 @@ _winapi_WriteFile(PyObject *module, PyObject **args, Py_ssize_t nargs, PyObject exit: return return_value; } -/*[clinic end generated code: output=46d6382a6662c4a9 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=6c5cf8865d381c70 input=a9049054013a1b77]*/ From webhook-mailer at python.org Fri Sep 7 11:20:46 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 15:20:46 -0000 Subject: [Python-checkins] bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099) Message-ID: https://github.com/python/cpython/commit/012f5b968a738b15ae9b40c499a1c0778b0615a9 commit: 012f5b968a738b15ae9b40c499a1c0778b0615a9 branch: master author: Victor Stinner committer: GitHub date: 2018-09-07T17:20:42+02:00 summary: bpo-34605, libregrtest: Rename --slaveargs to --worker-args (GH-9099) Rename also run_tests_slave() to run_tests_worker(). files: M Lib/test/libregrtest/cmdline.py M Lib/test/libregrtest/main.py M Lib/test/libregrtest/runtest_mp.py M Lib/test/test_regrtest.py diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py index dab17c3edf32..bd126b33c9bd 100644 --- a/Lib/test/libregrtest/cmdline.py +++ b/Lib/test/libregrtest/cmdline.py @@ -170,7 +170,7 @@ def _create_parser(): group.add_argument('--wait', action='store_true', help='wait for user input, e.g., allow a debugger ' 'to be attached') - group.add_argument('--slaveargs', metavar='ARGS') + group.add_argument('--worker-args', metavar='ARGS') group.add_argument('-S', '--start', metavar='START', help='the name of the test at which to start.' + more_details) diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index e262a7a172b9..a176db59c0fb 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -542,9 +542,9 @@ def _main(self, tests, kwargs): print(msg, file=sys.stderr, flush=True) sys.exit(2) - if self.ns.slaveargs is not None: - from test.libregrtest.runtest_mp import run_tests_slave - run_tests_slave(self.ns.slaveargs) + if self.ns.worker_args is not None: + from test.libregrtest.runtest_mp import run_tests_worker + run_tests_worker(self.ns.worker_args) if self.ns.wait: input("Press any key to continue...") diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py index 907451cf6311..1f07cfbd8cbe 100644 --- a/Lib/test/libregrtest/runtest_mp.py +++ b/Lib/test/libregrtest/runtest_mp.py @@ -24,23 +24,23 @@ def run_test_in_subprocess(testname, ns): - """Run the given test in a subprocess with --slaveargs. + """Run the given test in a subprocess with --worker-args. ns is the option Namespace parsed from command-line arguments. regrtest - is invoked in a subprocess with the --slaveargs argument; when the + is invoked in a subprocess with the --worker-args argument; when the subprocess exits, its return code, stdout and stderr are returned as a 3-tuple. """ from subprocess import Popen, PIPE ns_dict = vars(ns) - slaveargs = (ns_dict, testname) - slaveargs = json.dumps(slaveargs) + worker_args = (ns_dict, testname) + worker_args = json.dumps(worker_args) cmd = [sys.executable, *support.args_from_interpreter_flags(), '-u', # Unbuffered stdout and stderr '-m', 'test.regrtest', - '--slaveargs', slaveargs] + '--worker-args', worker_args] if ns.pgo: cmd += ['--pgo'] @@ -58,8 +58,8 @@ def run_test_in_subprocess(testname, ns): return retcode, stdout, stderr -def run_tests_slave(slaveargs): - ns_dict, testname = json.loads(slaveargs) +def run_tests_worker(worker_args): + ns_dict, testname = json.loads(worker_args) ns = types.SimpleNamespace(**ns_dict) setup_tests(ns) diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index af332ad15d92..f923311f3f20 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -67,10 +67,10 @@ def test_wait(self): ns = libregrtest._parse_args(['--wait']) self.assertTrue(ns.wait) - def test_slaveargs(self): - ns = libregrtest._parse_args(['--slaveargs', '[[], {}]']) - self.assertEqual(ns.slaveargs, '[[], {}]') - self.checkError(['--slaveargs'], 'expected one argument') + def test_worker_args(self): + ns = libregrtest._parse_args(['--worker-args', '[[], {}]']) + self.assertEqual(ns.worker_args, '[[], {}]') + self.checkError(['--worker-args'], 'expected one argument') def test_start(self): for opt in '-S', '--start': From webhook-mailer at python.org Fri Sep 7 11:30:36 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 15:30:36 -0000 Subject: [Python-checkins] bpo-34605: Avoid master/slave terms (GH-9101) Message-ID: https://github.com/python/cpython/commit/5e922658fb55734bf8b4c6246033ea93af172ff7 commit: 5e922658fb55734bf8b4c6246033ea93af172ff7 branch: master author: Victor Stinner committer: GitHub date: 2018-09-07T17:30:33+02:00 summary: bpo-34605: Avoid master/slave terms (GH-9101) * Replace "master process" with "parent process" * Replace "master option mappings" with "main option mappings" * Replace "master pattern object" with "main pattern object" * ssl: replace "master" with "server" * And some other similar changes files: M Doc/library/gc.rst M Doc/library/multiprocessing.rst M Lib/distutils/command/install.py M Lib/optparse.py M Lib/sre_parse.py M Lib/test/test_ssl.py M Tools/README diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 153d8fb70456..722a0e804314 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -181,7 +181,7 @@ The :mod:`gc` module provides the following functions: fork() call to make the gc copy-on-write friendly or to speed up collection. Also collection before a POSIX fork() call may free pages for future allocation which can cause copy-on-write too so it's advised to disable gc - in master process and freeze before fork and enable gc in child process. + in parent process and freeze before fork and enable gc in child process. .. versionadded:: 3.7 diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 504f3a1c3c33..a7d26f9b4554 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -435,7 +435,7 @@ process which created it. (If you try this it will actually output three full tracebacks interleaved in a semi-random fashion, and then you may have to - stop the master process somehow.) + stop the parent process somehow.) Reference diff --git a/Lib/distutils/command/install.py b/Lib/distutils/command/install.py index 0258d3deae94..41bf4bb9fb22 100644 --- a/Lib/distutils/command/install.py +++ b/Lib/distutils/command/install.py @@ -223,7 +223,7 @@ def initialize_options(self): def finalize_options(self): """Finalizes options.""" - # This method (and its pliant slaves, like 'finalize_unix()', + # This method (and its pliant childs, like 'finalize_unix()', # 'finalize_other()', and 'select_scheme()') is where the default # installation directories for modules, extension modules, and # anything else we care to install from a Python module diff --git a/Lib/optparse.py b/Lib/optparse.py index e8ac1e156a2b..1c450c6fcbe3 100644 --- a/Lib/optparse.py +++ b/Lib/optparse.py @@ -929,7 +929,7 @@ def __init__(self, option_class, conflict_handler, description): self.set_description(description) def _create_option_mappings(self): - # For use by OptionParser constructor -- create the master + # For use by OptionParser constructor -- create the main # option mappings used by this OptionParser and all # OptionGroups that it owns. self._short_opt = {} # single letter -> Option instance diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py index 7a172ff2fb14..f414402f9379 100644 --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -72,7 +72,7 @@ class Verbose(Exception): pass class Pattern: - # master pattern object. keeps track of global attributes + # main pattern object. keeps track of global attributes def __init__(self): self.flags = 0 self.groupdict = {} diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 2cb4ad4a36d4..e120a2f9adf2 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -3494,7 +3494,7 @@ def test_no_shared_ciphers(self): client_context, server_context, hostname = testing_context() # OpenSSL enables all TLS 1.3 ciphers, enforce TLS 1.2 for test client_context.options |= ssl.OP_NO_TLSv1_3 - # Force different suites on client and master + # Force different suites on client and server client_context.set_ciphers("AES128") server_context.set_ciphers("AES256") with ThreadedEchoServer(context=server_context) as server: diff --git a/Tools/README b/Tools/README index 35528811fee9..6c5fb2081812 100644 --- a/Tools/README +++ b/Tools/README @@ -1,7 +1,7 @@ This directory contains a number of Python programs that are useful while building or extending Python. -buildbot Batchfiles for running on Windows buildslaves. +buildbot Batchfiles for running on Windows buildbot workers. ccbench A Python threads-based concurrency benchmark. (*) From webhook-mailer at python.org Fri Sep 7 11:44:27 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Fri, 07 Sep 2018 15:44:27 -0000 Subject: [Python-checkins] bpo-20104: Add flag capabilities to posix_spawn (GH-6693) Message-ID: https://github.com/python/cpython/commit/254a4663d8c5970ae2928185c50ebaa6c7e62c80 commit: 254a4663d8c5970ae2928185c50ebaa6c7e62c80 branch: master author: Pablo Galindo committer: GitHub date: 2018-09-07T16:44:24+01:00 summary: bpo-20104: Add flag capabilities to posix_spawn (GH-6693) Implement the "attributes objects" parameter of `os.posix_spawn` to complete the implementation and fully cover the underlying API. files: A Misc/NEWS.d/next/Core and Builtins/2018-05-05-23-26-58.bpo-20104.tDBciE.rst M Doc/library/os.rst M Lib/test/test_posix.py M Modules/clinic/posixmodule.c.h M Modules/posixmodule.c diff --git a/Doc/library/os.rst b/Doc/library/os.rst index df136da02cb1..b8d6fffb303b 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3394,7 +3394,9 @@ written in Python, such as a mail server's external command delivery program. subprocesses. -.. function:: posix_spawn(path, argv, env, file_actions=None) +.. function:: posix_spawn(path, argv, env, file_actions=None, /, *, \ + setpgroup=None, resetids=False, setsigmask=(), \ + setsigdef=(), scheduler=None) Wraps the :c:func:`posix_spawn` C library API for use from Python. @@ -3432,6 +3434,36 @@ written in Python, such as a mail server's external command delivery program. :c:func:`posix_spawn_file_actions_adddup2` API calls used to prepare for the :c:func:`posix_spawn` call itself. + The *setpgroup* argument will set the process group of the child to the value + specified. If the value specified is 0, the child's process group ID will be + made the same as its process ID. If the value of *setpgroup* is not set, the + child will inherit the parent's process group ID. This argument corresponds + to the C library :c:data:`POSIX_SPAWN_SETPGROUP` flag. + + If the *resetids* argument is ``True`` it will reset the effective UID and + GID of the child to the real UID and GID of the parent process. If the + argument is ``False``, then the child retains the effective UID and GID of + the parent. In either case, if the set-user-ID and set-group-ID permission + bits are enabled on the executable file, their effect will override the + setting of the effective UID and GID. This argument corresponds to the C + library :c:data:`POSIX_SPAWN_RESETIDS` flag. + + The *setsigmask* argument will set the signal mask to the signal set + specified. If the parameter is not used, then the child inherits the + parent's signal mask. This argument corresponds to the C library + :c:data:`POSIX_SPAWN_SETSIGMASK` flag. + + The *sigdef* argument will reset the disposition of all signals in the set + specified. This argument corresponds to the C library + :c:data:`POSIX_SPAWN_SETSIGDEF` flag. + + The *scheduler* argument must be a tuple containing the (optional) scheduler + policy and an instance of :class:`sched_param` with the scheduler parameters. + A value of ``None`` in the place of the scheduler policy indicates that is + not being provided. This argument is a combination of the C library + :c:data:`POSIX_SPAWN_SETSCHEDPARAM` and :c:data:`POSIX_SPAWN_SETSCHEDULER` + flags. + .. versionadded:: 3.7 diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index ac18aa32528d..dafe9c1350e4 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -8,6 +8,7 @@ import errno import sys +import signal import time import os import platform @@ -16,6 +17,7 @@ import tempfile import unittest import warnings +import textwrap _DUMMY_SYMLINK = os.path.join(tempfile.gettempdir(), support.TESTFN + '-dummy-symlink') @@ -1540,6 +1542,147 @@ def test_empty_file_actions(self): ) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + def test_resetids_explicit_default(self): + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', 'pass'], + os.environ, + resetids=False + ) + self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + + def test_resetids(self): + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', 'pass'], + os.environ, + resetids=True + ) + self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + + def test_resetids_wrong_type(self): + with self.assertRaises(TypeError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, resetids=None) + + def test_setpgroup(self): + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', 'pass'], + os.environ, + setpgroup=os.getpgrp() + ) + self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + + def test_setpgroup_wrong_type(self): + with self.assertRaises(TypeError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setpgroup="023") + + @unittest.skipUnless(hasattr(signal, 'pthread_sigmask'), + 'need signal.pthread_sigmask()') + def test_setsigmask(self): + code = textwrap.dedent("""\ + import _testcapi, signal + _testcapi.raise_signal(signal.SIGUSR1)""") + + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', code], + os.environ, + setsigmask=[signal.SIGUSR1] + ) + self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + + def test_setsigmask_wrong_type(self): + with self.assertRaises(TypeError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setsigmask=34) + with self.assertRaises(TypeError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setsigmask=["j"]) + with self.assertRaises(ValueError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setsigmask=[signal.NSIG, + signal.NSIG+1]) + + @unittest.skipUnless(hasattr(signal, 'pthread_sigmask'), + 'need signal.pthread_sigmask()') + def test_setsigdef(self): + original_handler = signal.signal(signal.SIGUSR1, signal.SIG_IGN) + code = textwrap.dedent("""\ + import _testcapi, signal + _testcapi.raise_signal(signal.SIGUSR1)""") + try: + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', code], + os.environ, + setsigdef=[signal.SIGUSR1] + ) + finally: + signal.signal(signal.SIGUSR1, original_handler) + + pid2, status = os.waitpid(pid, 0) + self.assertEqual(pid2, pid) + self.assertTrue(os.WIFSIGNALED(status), status) + self.assertEqual(os.WTERMSIG(status), signal.SIGUSR1) + + def test_setsigdef_wrong_type(self): + with self.assertRaises(TypeError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setsigdef=34) + with self.assertRaises(TypeError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setsigdef=["j"]) + with self.assertRaises(ValueError): + posix.posix_spawn(sys.executable, + [sys.executable, "-c", "pass"], + os.environ, setsigdef=[signal.NSIG, signal.NSIG+1]) + + @unittest.skipUnless(hasattr(posix, 'sched_setscheduler'), "can't change scheduler") + def test_setscheduler_only_param(self): + policy = os.sched_getscheduler(0) + priority = os.sched_get_priority_min(policy) + code = textwrap.dedent(f"""\ + import os + if os.sched_getscheduler(0) != {policy}: + os.exit(101) + if os.sched_getparam(0).sched_priority != {priority}: + os.exit(102)""") + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', code], + os.environ, + scheduler=(None, os.sched_param(priority)) + ) + self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + + @unittest.skipUnless(hasattr(posix, 'sched_setscheduler'), "can't change scheduler") + def test_setscheduler_with_policy(self): + policy = os.sched_getscheduler(0) + priority = os.sched_get_priority_min(policy) + code = textwrap.dedent(f"""\ + import os + if os.sched_getscheduler(0) != {policy}: + os.exit(101) + if os.sched_getparam(0).sched_priority != {priority}: + os.exit(102)""") + pid = posix.posix_spawn( + sys.executable, + [sys.executable, '-c', code], + os.environ, + scheduler=(policy, os.sched_param(priority)) + ) + self.assertEqual(os.waitpid(pid, 0), (pid, 0)) + def test_multiple_file_actions(self): file_actions = [ (os.POSIX_SPAWN_OPEN, 3, os.path.realpath(__file__), os.O_RDONLY, 0), diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-05-05-23-26-58.bpo-20104.tDBciE.rst b/Misc/NEWS.d/next/Core and Builtins/2018-05-05-23-26-58.bpo-20104.tDBciE.rst new file mode 100644 index 000000000000..1d725ba7eb71 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-05-05-23-26-58.bpo-20104.tDBciE.rst @@ -0,0 +1,2 @@ +Added support for the `setpgroup`, `resetids`, `setsigmask`, `setsigdef` and +`scheduler` parameters of `posix_spawn`. Patch by Pablo Galindo. diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 7a2188504ac5..f7767c4af0bb 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -1730,7 +1730,9 @@ os_execve(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *k #if defined(HAVE_POSIX_SPAWN) PyDoc_STRVAR(os_posix_spawn__doc__, -"posix_spawn($module, path, argv, env, file_actions=None, /)\n" +"posix_spawn($module, path, argv, env, file_actions=None, /, *,\n" +" setpgroup=None, resetids=False, setsigmask=(),\n" +" setsigdef=(), scheduler=None)\n" "--\n" "\n" "Execute the program specified by path in a new process.\n" @@ -1742,29 +1744,48 @@ PyDoc_STRVAR(os_posix_spawn__doc__, " env\n" " Dictionary of strings mapping to strings.\n" " file_actions\n" -" A sequence of file action tuples."); +" A sequence of file action tuples.\n" +" setpgroup\n" +" The pgroup to use with the POSIX_SPAWN_SETPGROUP flag.\n" +" resetids\n" +" If the value is `True` the POSIX_SPAWN_RESETIDS will be activated.\n" +" setsigmask\n" +" The sigmask to use with the POSIX_SPAWN_SETSIGMASK flag.\n" +" setsigdef\n" +" The sigmask to use with the POSIX_SPAWN_SETSIGDEF flag.\n" +" scheduler\n" +" A tuple with the scheduler policy (optional) and parameters."); #define OS_POSIX_SPAWN_METHODDEF \ - {"posix_spawn", (PyCFunction)os_posix_spawn, METH_FASTCALL, os_posix_spawn__doc__}, + {"posix_spawn", (PyCFunction)os_posix_spawn, METH_FASTCALL|METH_KEYWORDS, os_posix_spawn__doc__}, static PyObject * os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, - PyObject *env, PyObject *file_actions); + PyObject *env, PyObject *file_actions, + PyObject *setpgroup, int resetids, PyObject *setsigmask, + PyObject *setsigdef, PyObject *scheduler); static PyObject * -os_posix_spawn(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +os_posix_spawn(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; + static const char * const _keywords[] = {"", "", "", "", "setpgroup", "resetids", "setsigmask", "setsigdef", "scheduler", NULL}; + static _PyArg_Parser _parser = {"O&OO|O$OiOOO:posix_spawn", _keywords, 0}; path_t path = PATH_T_INITIALIZE("posix_spawn", "path", 0, 0); PyObject *argv; PyObject *env; PyObject *file_actions = Py_None; + PyObject *setpgroup = NULL; + int resetids = 0; + PyObject *setsigmask = NULL; + PyObject *setsigdef = NULL; + PyObject *scheduler = NULL; - if (!_PyArg_ParseStack(args, nargs, "O&OO|O:posix_spawn", - path_converter, &path, &argv, &env, &file_actions)) { + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + path_converter, &path, &argv, &env, &file_actions, &setpgroup, &resetids, &setsigmask, &setsigdef, &scheduler)) { goto exit; } - return_value = os_posix_spawn_impl(module, &path, argv, env, file_actions); + return_value = os_posix_spawn_impl(module, &path, argv, env, file_actions, setpgroup, resetids, setsigmask, setsigdef, scheduler); exit: /* Cleanup for path */ @@ -6627,4 +6648,4 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject #ifndef OS_GETRANDOM_METHODDEF #define OS_GETRANDOM_METHODDEF #endif /* !defined(OS_GETRANDOM_METHODDEF) */ -/*[clinic end generated code: output=47fb6a3e88cba6d9 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=ef78384ae88712e1 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 7f13735eadc8..4a8a8d70ae42 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -5177,6 +5177,114 @@ enum posix_spawn_file_actions_identifier { POSIX_SPAWN_DUP2 }; +static int +convert_sched_param(PyObject *param, struct sched_param *res); + +static int +parse_posix_spawn_flags(PyObject *setpgroup, int resetids, PyObject *setsigmask, + PyObject *setsigdef, PyObject *scheduler, + posix_spawnattr_t *attrp) +{ + long all_flags = 0; + + errno = posix_spawnattr_init(attrp); + if (errno) { + posix_error(); + return -1; + } + + if (setpgroup) { + pid_t pgid = PyLong_AsPid(setpgroup); + if (pgid == (pid_t)-1 && PyErr_Occurred()) { + goto fail; + } + errno = posix_spawnattr_setpgroup(attrp, pgid); + if (errno) { + posix_error(); + goto fail; + } + all_flags |= POSIX_SPAWN_SETPGROUP; + } + + if (resetids) { + all_flags |= POSIX_SPAWN_RESETIDS; + } + + if (setsigmask) { + sigset_t set; + if (!_Py_Sigset_Converter(setsigmask, &set)) { + goto fail; + } + errno = posix_spawnattr_setsigmask(attrp, &set); + if (errno) { + posix_error(); + goto fail; + } + all_flags |= POSIX_SPAWN_SETSIGMASK; + } + + if (setsigdef) { + sigset_t set; + if (!_Py_Sigset_Converter(setsigdef, &set)) { + goto fail; + } + errno = posix_spawnattr_setsigdefault(attrp, &set); + if (errno) { + posix_error(); + goto fail; + } + all_flags |= POSIX_SPAWN_SETSIGDEF; + } + + if (scheduler) { +#ifdef POSIX_SPAWN_SETSCHEDULER + PyObject *py_schedpolicy; + struct sched_param schedparam; + + if (!PyArg_ParseTuple(scheduler, "OO&" + ";A scheduler tuple must have two elements", + &py_schedpolicy, convert_sched_param, &schedparam)) { + goto fail; + } + if (py_schedpolicy != Py_None) { + int schedpolicy = _PyLong_AsInt(py_schedpolicy); + + if (schedpolicy == -1 && PyErr_Occurred()) { + goto fail; + } + errno = posix_spawnattr_setschedpolicy(attrp, schedpolicy); + if (errno) { + posix_error(); + goto fail; + } + all_flags |= POSIX_SPAWN_SETSCHEDULER; + } + errno = posix_spawnattr_setschedparam(attrp, &schedparam); + if (errno) { + posix_error(); + goto fail; + } + all_flags |= POSIX_SPAWN_SETSCHEDPARAM; +#else + PyErr_SetString(PyExc_NotImplementedError, + "The scheduler option is not supported in this system."); + goto fail; +#endif + } + + errno = posix_spawnattr_setflags(attrp, all_flags); + if (errno) { + posix_error(); + goto fail; + } + + return 0; + +fail: + (void)posix_spawnattr_destroy(attrp); + return -1; +} + static int parse_file_actions(PyObject *file_actions, posix_spawn_file_actions_t *file_actionsp, @@ -5277,6 +5385,7 @@ parse_file_actions(PyObject *file_actions, } Py_DECREF(file_action); } + Py_DECREF(seq); return 0; @@ -5299,19 +5408,33 @@ os.posix_spawn file_actions: object = None A sequence of file action tuples. / - + * + setpgroup: object = NULL + The pgroup to use with the POSIX_SPAWN_SETPGROUP flag. + resetids: bool(accept={int}) = False + If the value is `True` the POSIX_SPAWN_RESETIDS will be activated. + setsigmask: object(c_default='NULL') = () + The sigmask to use with the POSIX_SPAWN_SETSIGMASK flag. + setsigdef: object(c_default='NULL') = () + The sigmask to use with the POSIX_SPAWN_SETSIGDEF flag. + scheduler: object = NULL + A tuple with the scheduler policy (optional) and parameters. Execute the program specified by path in a new process. [clinic start generated code]*/ static PyObject * os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, - PyObject *env, PyObject *file_actions) -/*[clinic end generated code: output=d023521f541c709c input=a3db1021d33230dc]*/ + PyObject *env, PyObject *file_actions, + PyObject *setpgroup, int resetids, PyObject *setsigmask, + PyObject *setsigdef, PyObject *scheduler) +/*[clinic end generated code: output=45dfa4c515d09f2c input=2d7a7578430a90f0]*/ { EXECV_CHAR **argvlist = NULL; EXECV_CHAR **envlist = NULL; posix_spawn_file_actions_t file_actions_buf; posix_spawn_file_actions_t *file_actionsp = NULL; + posix_spawnattr_t attr; + posix_spawnattr_t *attrp = NULL; Py_ssize_t argc, envc; PyObject *result = NULL; PyObject *temp_buffer = NULL; @@ -5373,9 +5496,15 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, file_actionsp = &file_actions_buf; } + if (parse_posix_spawn_flags(setpgroup, resetids, setsigmask, + setsigdef, scheduler, &attr)) { + goto exit; + } + attrp = &attr; + _Py_BEGIN_SUPPRESS_IPH err_code = posix_spawn(&pid, path->narrow, - file_actionsp, NULL, argvlist, envlist); + file_actionsp, attrp, argvlist, envlist); _Py_END_SUPPRESS_IPH if (err_code) { errno = err_code; @@ -5388,6 +5517,9 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, if (file_actionsp) { (void)posix_spawn_file_actions_destroy(file_actionsp); } + if (attrp) { + (void)posix_spawnattr_destroy(attrp); + } if (envlist) { free_string_array(envlist, envc); } From webhook-mailer at python.org Fri Sep 7 12:01:04 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 16:01:04 -0000 Subject: [Python-checkins] bpo-34595: Add %T format to PyUnicode_FromFormatV() (GH-9080) Message-ID: https://github.com/python/cpython/commit/886483e2b9bbabf60ab769683269b873381dd5ee commit: 886483e2b9bbabf60ab769683269b873381dd5ee branch: master author: Victor Stinner committer: GitHub date: 2018-09-07T18:00:58+02:00 summary: bpo-34595: Add %T format to PyUnicode_FromFormatV() (GH-9080) * Add %T format to PyUnicode_FromFormatV(), and so to PyUnicode_FromFormat() and PyErr_Format(), to format an object type name: equivalent to "%s" with Py_TYPE(obj)->tp_name. * Replace Py_TYPE(obj)->tp_name with %T format in unicodeobject.c. * Add unit test on %T format. * Rename unicode_fromformat_write_cstr() to unicode_fromformat_write_utf8(), to make the intent more explicit. files: A Misc/NEWS.d/next/C API/2018-09-06-11-17-49.bpo-34595.Hkz62y.rst M Doc/c-api/unicode.rst M Lib/test/test_unicode.py M Objects/unicodeobject.c diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 92e22b16a4ef..66b1efc60fde 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -519,6 +519,9 @@ APIs: | :attr:`%R` | PyObject\* | The result of calling | | | | :c:func:`PyObject_Repr`. | +-------------------+---------------------+--------------------------------+ + | :attr:`%T` | PyObject\* | Object type name, equivalent | + | | | to ``Py_TYPE(op)->tp_name``. | + +-------------------+---------------------+--------------------------------+ An unrecognized format character causes all the rest of the format string to be copied as-is to the result string, and any extra arguments discarded. @@ -543,6 +546,9 @@ APIs: Support width and precision formatter for ``"%s"``, ``"%A"``, ``"%U"``, ``"%V"``, ``"%S"``, ``"%R"`` added. + .. versionchanged:: 3.7 + Support for ``"%T"`` (object type name) added. + .. c:function:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py index fb7bb2d523fe..73111f1c24a2 100644 --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2655,6 +2655,10 @@ def check_format(expected, format, *args): check_format(r"%A:'abc\xe9\uabcd\U0010ffff'", b'%%A:%A', 'abc\xe9\uabcd\U0010ffff') + # test %T (object type name) + check_format(r"type name: str", + b'type name: %T', 'text') + # test %V check_format('repr=abc', b'repr=%V', 'abc', b'xyz') diff --git a/Misc/NEWS.d/next/C API/2018-09-06-11-17-49.bpo-34595.Hkz62y.rst b/Misc/NEWS.d/next/C API/2018-09-06-11-17-49.bpo-34595.Hkz62y.rst new file mode 100644 index 000000000000..c054a8e229f5 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2018-09-06-11-17-49.bpo-34595.Hkz62y.rst @@ -0,0 +1,4 @@ +:c:func:`PyUnicode_FromFormatV`: add ``%T`` format to +:c:func:`PyUnicode_FromFormatV`, and so to :c:func:`PyUnicode_FromFormat` +and :c:func:`PyErr_Format`, to format an object type name: equivalent to +"%s" with ``Py_TYPE(obj)->tp_name``. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index a797f838eb41..3e61c9c37010 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -768,8 +768,7 @@ ensure_unicode(PyObject *obj) { if (!PyUnicode_Check(obj)) { PyErr_Format(PyExc_TypeError, - "must be str, not %.100s", - Py_TYPE(obj)->tp_name); + "must be str, not %T", obj); return -1; } return PyUnicode_READY(obj); @@ -2530,7 +2529,7 @@ unicode_fromformat_write_str(_PyUnicodeWriter *writer, PyObject *str, } static int -unicode_fromformat_write_cstr(_PyUnicodeWriter *writer, const char *str, +unicode_fromformat_write_utf8(_PyUnicodeWriter *writer, const char *str, Py_ssize_t width, Py_ssize_t precision) { /* UTF-8 */ @@ -2747,7 +2746,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, { /* UTF-8 */ const char *s = va_arg(*vargs, const char*); - if (unicode_fromformat_write_cstr(writer, s, width, precision) < 0) + if (unicode_fromformat_write_utf8(writer, s, width, precision) < 0) return NULL; break; } @@ -2773,7 +2772,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, } else { assert(str != NULL); - if (unicode_fromformat_write_cstr(writer, str, width, precision) < 0) + if (unicode_fromformat_write_utf8(writer, str, width, precision) < 0) return NULL; } break; @@ -2827,6 +2826,17 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, break; } + case 'T': + { + /* Object type name (tp_name) */ + PyObject *obj = va_arg(*vargs, PyObject *); + PyTypeObject *type = Py_TYPE(obj); + const char *type_name = type->tp_name; + if (unicode_fromformat_write_utf8(writer, type_name, -1, -1) < 0) { + return NULL; + } + break; + } case '%': if (_PyUnicodeWriter_WriteCharInline(writer, '%') < 0) return NULL; @@ -3024,8 +3034,7 @@ PyUnicode_FromObject(PyObject *obj) return _PyUnicode_Copy(obj); } PyErr_Format(PyExc_TypeError, - "Can't convert '%.100s' object to str implicitly", - Py_TYPE(obj)->tp_name); + "Can't convert '%T' object to str implicitly", obj); return NULL; } @@ -3061,8 +3070,8 @@ PyUnicode_FromEncodedObject(PyObject *obj, /* Retrieve a bytes buffer view through the PEP 3118 buffer interface */ if (PyObject_GetBuffer(obj, &buffer, PyBUF_SIMPLE) < 0) { PyErr_Format(PyExc_TypeError, - "decoding to str: need a bytes-like object, %.80s found", - Py_TYPE(obj)->tp_name); + "decoding to str: need a bytes-like object, %T found", + obj); return NULL; } @@ -3192,10 +3201,9 @@ PyUnicode_Decode(const char *s, goto onError; if (!PyUnicode_Check(unicode)) { PyErr_Format(PyExc_TypeError, - "'%.400s' decoder returned '%.400s' instead of 'str'; " + "'%.400s' decoder returned '%T' instead of 'str'; " "use codecs.decode() to decode to arbitrary types", - encoding, - Py_TYPE(unicode)->tp_name); + encoding, unicode); Py_DECREF(unicode); goto onError; } @@ -3255,10 +3263,9 @@ PyUnicode_AsDecodedUnicode(PyObject *unicode, goto onError; if (!PyUnicode_Check(v)) { PyErr_Format(PyExc_TypeError, - "'%.400s' decoder returned '%.400s' instead of 'str'; " + "'%.400s' decoder returned '%T' instead of 'str'; " "use codecs.decode() to decode to arbitrary types", - encoding, - Py_TYPE(unicode)->tp_name); + encoding, unicode); Py_DECREF(v); goto onError; } @@ -3489,10 +3496,9 @@ PyUnicode_AsEncodedString(PyObject *unicode, } PyErr_Format(PyExc_TypeError, - "'%.400s' encoder returned '%.400s' instead of 'bytes'; " + "'%.400s' encoder returned '%T' instead of 'bytes'; " "use codecs.encode() to encode to arbitrary types", - encoding, - Py_TYPE(v)->tp_name); + encoding, v); Py_DECREF(v); return NULL; } @@ -3523,10 +3529,9 @@ PyUnicode_AsEncodedUnicode(PyObject *unicode, goto onError; if (!PyUnicode_Check(v)) { PyErr_Format(PyExc_TypeError, - "'%.400s' encoder returned '%.400s' instead of 'str'; " + "'%.400s' encoder returned '%T' instead of 'str'; " "use codecs.encode() to encode to arbitrary types", - encoding, - Py_TYPE(v)->tp_name); + encoding, v); Py_DECREF(v); goto onError; } @@ -3698,9 +3703,11 @@ PyUnicode_FSDecoder(PyObject* arg, void* addr) if (!PyBytes_Check(path) && PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "path should be string, bytes, or os.PathLike, not %.200s", - Py_TYPE(arg)->tp_name)) { - Py_DECREF(path); + "path should be string, bytes, " + "or os.PathLike, not %T", + arg)) + { + Py_DECREF(path); return 0; } path_bytes = PyBytes_FromObject(path); @@ -3717,8 +3724,8 @@ PyUnicode_FSDecoder(PyObject* arg, void* addr) } else { PyErr_Format(PyExc_TypeError, - "path should be string, bytes, or os.PathLike, not %.200s", - Py_TYPE(arg)->tp_name); + "path should be string, bytes, or os.PathLike, not %T", + arg); Py_DECREF(path); return 0; } @@ -9886,9 +9893,8 @@ _PyUnicode_JoinArray(PyObject *separator, PyObject *const *items, Py_ssize_t seq else { if (!PyUnicode_Check(separator)) { PyErr_Format(PyExc_TypeError, - "separator: expected str instance," - " %.80s found", - Py_TYPE(separator)->tp_name); + "separator: expected str instance, %T found", + separator); goto onError; } if (PyUnicode_READY(separator)) @@ -9919,9 +9925,8 @@ _PyUnicode_JoinArray(PyObject *separator, PyObject *const *items, Py_ssize_t seq item = items[i]; if (!PyUnicode_Check(item)) { PyErr_Format(PyExc_TypeError, - "sequence item %zd: expected str instance," - " %.80s found", - i, Py_TYPE(item)->tp_name); + "sequence item %zd: expected str instance, %T found", + i, item); goto onError; } if (PyUnicode_READY(item) == -1) @@ -10736,7 +10741,7 @@ convert_uc(PyObject *obj, void *addr) if (!PyUnicode_Check(obj)) { PyErr_Format(PyExc_TypeError, "The fill character must be a unicode character, " - "not %.100s", Py_TYPE(obj)->tp_name); + "not %T", obj); return 0; } if (PyUnicode_READY(obj) < 0) @@ -11142,8 +11147,8 @@ PyUnicode_Contains(PyObject *str, PyObject *substr) if (!PyUnicode_Check(substr)) { PyErr_Format(PyExc_TypeError, - "'in ' requires string as left operand, not %.100s", - Py_TYPE(substr)->tp_name); + "'in ' requires string as left operand, not %T", + substr); return -1; } if (PyUnicode_READY(substr) == -1) @@ -12848,9 +12853,7 @@ unicode_split_impl(PyObject *self, PyObject *sep, Py_ssize_t maxsplit) if (PyUnicode_Check(sep)) return split(self, sep, maxsplit); - PyErr_Format(PyExc_TypeError, - "must be str or None, not %.100s", - Py_TYPE(sep)->tp_name); + PyErr_Format(PyExc_TypeError, "must be str or None, not %T", sep); return NULL; } @@ -13036,9 +13039,7 @@ unicode_rsplit_impl(PyObject *self, PyObject *sep, Py_ssize_t maxsplit) if (PyUnicode_Check(sep)) return rsplit(self, sep, maxsplit); - PyErr_Format(PyExc_TypeError, - "must be str or None, not %.100s", - Py_TYPE(sep)->tp_name); + PyErr_Format(PyExc_TypeError, "must be str or None, not %T", sep); return NULL; } @@ -13333,8 +13334,8 @@ unicode_startswith(PyObject *self, if (!PyUnicode_Check(substring)) { PyErr_Format(PyExc_TypeError, "tuple for startswith must only contain str, " - "not %.100s", - Py_TYPE(substring)->tp_name); + "not %T", + substring); return NULL; } result = tailmatch(self, substring, start, end, -1); @@ -13350,7 +13351,7 @@ unicode_startswith(PyObject *self, if (!PyUnicode_Check(subobj)) { PyErr_Format(PyExc_TypeError, "startswith first arg must be str or " - "a tuple of str, not %.100s", Py_TYPE(subobj)->tp_name); + "a tuple of str, not %T", subobj); return NULL; } result = tailmatch(self, subobj, start, end, -1); @@ -13387,8 +13388,8 @@ unicode_endswith(PyObject *self, if (!PyUnicode_Check(substring)) { PyErr_Format(PyExc_TypeError, "tuple for endswith must only contain str, " - "not %.100s", - Py_TYPE(substring)->tp_name); + "not %T", + substring); return NULL; } result = tailmatch(self, substring, start, end, +1); @@ -13403,7 +13404,7 @@ unicode_endswith(PyObject *self, if (!PyUnicode_Check(subobj)) { PyErr_Format(PyExc_TypeError, "endswith first arg must be str or " - "a tuple of str, not %.100s", Py_TYPE(subobj)->tp_name); + "a tuple of str, not %T", subobj); return NULL; } result = tailmatch(self, subobj, start, end, +1); @@ -14313,15 +14314,13 @@ mainformatlong(PyObject *v, case 'x': case 'X': PyErr_Format(PyExc_TypeError, - "%%%c format: an integer is required, " - "not %.200s", - type, Py_TYPE(v)->tp_name); + "%%%c format: an integer is required, not %T", + type, v); break; default: PyErr_Format(PyExc_TypeError, - "%%%c format: a number is required, " - "not %.200s", - type, Py_TYPE(v)->tp_name); + "%%%c format: a number is required, not %T", + type, v); break; } return -1; From webhook-mailer at python.org Fri Sep 7 12:13:14 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 16:13:14 -0000 Subject: [Python-checkins] bpo-34605: childs => children (GH-9102) Message-ID: https://github.com/python/cpython/commit/7e610bcdf128f61b925654e4fa80fbac83537d0e commit: 7e610bcdf128f61b925654e4fa80fbac83537d0e branch: master author: Victor Stinner committer: GitHub date: 2018-09-07T18:13:10+02:00 summary: bpo-34605: childs => children (GH-9102) files: M Lib/distutils/command/install.py diff --git a/Lib/distutils/command/install.py b/Lib/distutils/command/install.py index 41bf4bb9fb22..a1d1a1ea37ad 100644 --- a/Lib/distutils/command/install.py +++ b/Lib/distutils/command/install.py @@ -223,7 +223,7 @@ def initialize_options(self): def finalize_options(self): """Finalizes options.""" - # This method (and its pliant childs, like 'finalize_unix()', + # This method (and its pliant children, like 'finalize_unix()', # 'finalize_other()', and 'select_scheme()') is where the default # installation directories for modules, extension modules, and # anything else we care to install from a Python module From webhook-mailer at python.org Fri Sep 7 12:17:13 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Fri, 07 Sep 2018 16:17:13 -0000 Subject: [Python-checkins] [3.6] bpo-34594: Don't hardcode errno values in the tests. (GH-9096) Message-ID: https://github.com/python/cpython/commit/9eeecfd5d03cfd6d5cef71de31eed7f65f0e658b commit: 9eeecfd5d03cfd6d5cef71de31eed7f65f0e658b branch: 3.6 author: Zackery Spytz committer: Benjamin Peterson date: 2018-09-07T09:17:08-07:00 summary: [3.6] bpo-34594: Don't hardcode errno values in the tests. (GH-9096) (cherry picked from commit b03c2c51909e3b5b5966d86a2829b5ddf2d496aa) files: A Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst M Lib/test/test_spwd.py diff --git a/Lib/test/test_spwd.py b/Lib/test/test_spwd.py index e893f3a847fd..07793c84c8e9 100644 --- a/Lib/test/test_spwd.py +++ b/Lib/test/test_spwd.py @@ -67,8 +67,6 @@ def test_getspnam_exception(self): spwd.getspnam(name) except KeyError as exc: self.skipTest("spwd entry %r doesn't exist: %s" % (name, exc)) - else: - self.assertEqual(str(cm.exception), '[Errno 13] Permission denied') if __name__ == "__main__": diff --git a/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst b/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst new file mode 100644 index 000000000000..c6026b0f717b --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst @@ -0,0 +1 @@ +Fix usage of hardcoded ``errno`` values in the tests. \ No newline at end of file From webhook-mailer at python.org Fri Sep 7 12:17:36 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Fri, 07 Sep 2018 16:17:36 -0000 Subject: [Python-checkins] [3.7] bpo-34594: Don't hardcode errno values in the tests. (GH-9094) Message-ID: https://github.com/python/cpython/commit/18d7dff1bb6f5ca7060b0b2e2a2e74493619178f commit: 18d7dff1bb6f5ca7060b0b2e2a2e74493619178f branch: 3.7 author: Zackery Spytz committer: Benjamin Peterson date: 2018-09-07T09:17:32-07:00 summary: [3.7] bpo-34594: Don't hardcode errno values in the tests. (GH-9094) (cherry picked from commit b03c2c51909e3b5b5966d86a2829b5ddf2d496aa) files: A Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst M Lib/test/test_spwd.py diff --git a/Lib/test/test_spwd.py b/Lib/test/test_spwd.py index e893f3a847fd..07793c84c8e9 100644 --- a/Lib/test/test_spwd.py +++ b/Lib/test/test_spwd.py @@ -67,8 +67,6 @@ def test_getspnam_exception(self): spwd.getspnam(name) except KeyError as exc: self.skipTest("spwd entry %r doesn't exist: %s" % (name, exc)) - else: - self.assertEqual(str(cm.exception), '[Errno 13] Permission denied') if __name__ == "__main__": diff --git a/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst b/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst new file mode 100644 index 000000000000..c6026b0f717b --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2018-09-05-23-50-21.bpo-34594.tqL-GS.rst @@ -0,0 +1 @@ +Fix usage of hardcoded ``errno`` values in the tests. \ No newline at end of file From webhook-mailer at python.org Fri Sep 7 13:10:43 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Sep 2018 17:10:43 -0000 Subject: [Python-checkins] bpo-34604: Fix possible mojibake in pwd.getpwnam() and grp.getgrnam() (GH-9098) Message-ID: https://github.com/python/cpython/commit/28658485a54ad5f9df52ecc12d9046269f1654ec commit: 28658485a54ad5f9df52ecc12d9046269f1654ec branch: master author: William Grzybowski committer: Victor Stinner date: 2018-09-07T19:10:39+02:00 summary: bpo-34604: Fix possible mojibake in pwd.getpwnam() and grp.getgrnam() (GH-9098) Pass the user/group name as Unicode to the formatting function, instead of always decoding a bytes string from UTF-8. files: A Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst M Modules/clinic/pwdmodule.c.h M Modules/grpmodule.c M Modules/pwdmodule.c diff --git a/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst b/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst new file mode 100644 index 000000000000..562a69124b3d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst @@ -0,0 +1,2 @@ +Fix possible mojibake in the error message of `pwd.getpwnam` and +`grp.getgrnam`. Patch by William Grzybowski. diff --git a/Modules/clinic/pwdmodule.c.h b/Modules/clinic/pwdmodule.c.h index f9e0644f264d..979dfdb07ee2 100644 --- a/Modules/clinic/pwdmodule.c.h +++ b/Modules/clinic/pwdmodule.c.h @@ -14,7 +14,7 @@ PyDoc_STRVAR(pwd_getpwuid__doc__, {"getpwuid", (PyCFunction)pwd_getpwuid, METH_O, pwd_getpwuid__doc__}, PyDoc_STRVAR(pwd_getpwnam__doc__, -"getpwnam($module, arg, /)\n" +"getpwnam($module, name, /)\n" "--\n" "\n" "Return the password database entry for the given user name.\n" @@ -25,18 +25,18 @@ PyDoc_STRVAR(pwd_getpwnam__doc__, {"getpwnam", (PyCFunction)pwd_getpwnam, METH_O, pwd_getpwnam__doc__}, static PyObject * -pwd_getpwnam_impl(PyObject *module, PyObject *arg); +pwd_getpwnam_impl(PyObject *module, PyObject *name); static PyObject * -pwd_getpwnam(PyObject *module, PyObject *arg_) +pwd_getpwnam(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; - PyObject *arg; + PyObject *name; - if (!PyArg_Parse(arg_, "U:getpwnam", &arg)) { + if (!PyArg_Parse(arg, "U:getpwnam", &name)) { goto exit; } - return_value = pwd_getpwnam_impl(module, arg); + return_value = pwd_getpwnam_impl(module, name); exit: return return_value; @@ -69,4 +69,4 @@ pwd_getpwall(PyObject *module, PyObject *Py_UNUSED(ignored)) #ifndef PWD_GETPWALL_METHODDEF #define PWD_GETPWALL_METHODDEF #endif /* !defined(PWD_GETPWALL_METHODDEF) */ -/*[clinic end generated code: output=fc41d8d88ec206d8 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=3c93120d6dd86905 input=a9049054013a1b77]*/ diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c index ffdfa1b6f9fb..74286ab3974d 100644 --- a/Modules/grpmodule.c +++ b/Modules/grpmodule.c @@ -245,7 +245,7 @@ grp_getgrnam_impl(PyObject *module, PyObject *name) PyErr_NoMemory(); } else { - PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %s", name_chars); + PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %S", name); } goto out; } diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c index 36192a570433..d15286dc10fc 100644 --- a/Modules/pwdmodule.c +++ b/Modules/pwdmodule.c @@ -189,7 +189,7 @@ pwd_getpwuid(PyObject *module, PyObject *uidobj) /*[clinic input] pwd.getpwnam - arg: unicode + name: unicode / Return the password database entry for the given user name. @@ -198,18 +198,18 @@ See `help(pwd)` for more on password database entries. [clinic start generated code]*/ static PyObject * -pwd_getpwnam_impl(PyObject *module, PyObject *arg) -/*[clinic end generated code: output=6abeee92430e43d2 input=d5f7e700919b02d3]*/ +pwd_getpwnam_impl(PyObject *module, PyObject *name) +/*[clinic end generated code: output=359ce1ddeb7a824f input=a6aeb5e3447fb9e0]*/ { - char *buf = NULL, *buf2 = NULL, *name; + char *buf = NULL, *buf2 = NULL, *name_chars; int nomem = 0; struct passwd *p; PyObject *bytes, *retval = NULL; - if ((bytes = PyUnicode_EncodeFSDefault(arg)) == NULL) + if ((bytes = PyUnicode_EncodeFSDefault(name)) == NULL) return NULL; /* check for embedded null bytes */ - if (PyBytes_AsStringAndSize(bytes, &name, NULL) == -1) + if (PyBytes_AsStringAndSize(bytes, &name_chars, NULL) == -1) goto out; #ifdef HAVE_GETPWNAM_R Py_BEGIN_ALLOW_THREADS @@ -229,7 +229,7 @@ pwd_getpwnam_impl(PyObject *module, PyObject *arg) break; } buf = buf2; - status = getpwnam_r(name, &pwd, buf, bufsize, &p); + status = getpwnam_r(name_chars, &pwd, buf, bufsize, &p); if (status != 0) { p = NULL; } @@ -245,7 +245,7 @@ pwd_getpwnam_impl(PyObject *module, PyObject *arg) Py_END_ALLOW_THREADS #else - p = getpwnam(name); + p = getpwnam(name_chars); #endif if (p == NULL) { if (nomem == 1) { @@ -253,7 +253,7 @@ pwd_getpwnam_impl(PyObject *module, PyObject *arg) } else { PyErr_Format(PyExc_KeyError, - "getpwnam(): name not found: %s", name); + "getpwnam(): name not found: %S", name); } goto out; } From webhook-mailer at python.org Fri Sep 7 18:02:59 2018 From: webhook-mailer at python.org (Berker Peksag) Date: Fri, 07 Sep 2018 22:02:59 -0000 Subject: [Python-checkins] bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) Message-ID: https://github.com/python/cpython/commit/4e519377b1b84c9414a360961276993d24198825 commit: 4e519377b1b84c9414a360961276993d24198825 branch: master author: Zackery Spytz committer: Berker Peksag date: 2018-09-08T01:02:56+03:00 summary: bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) files: M PC/_msi.c diff --git a/PC/_msi.c b/PC/_msi.c index 000d81f139f3..024b2d3c9fd3 100644 --- a/PC/_msi.c +++ b/PC/_msi.c @@ -330,6 +330,10 @@ msierror(int status) code = MsiRecordGetInteger(err, 1); /* XXX code */ if (MsiFormatRecord(0, err, res, &size) == ERROR_MORE_DATA) { res = malloc(size+1); + if (res == NULL) { + MsiCloseHandle(err); + return PyErr_NoMemory(); + } MsiFormatRecord(0, err, res, &size); res[size]='\0'; } @@ -560,6 +564,9 @@ summary_getproperty(msiobj* si, PyObject *args) &fval, sval, &ssize); if (status == ERROR_MORE_DATA) { sval = malloc(ssize); + if (sval == NULL) { + return PyErr_NoMemory(); + } status = MsiSummaryInfoGetProperty(si->h, field, &type, &ival, &fval, sval, &ssize); } From webhook-mailer at python.org Fri Sep 7 18:04:51 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Fri, 07 Sep 2018 22:04:51 -0000 Subject: [Python-checkins] bpo-34246: Use no mutable default args in smtplib (GH-8554) Message-ID: https://github.com/python/cpython/commit/d5fbe9b1a3d65ceeb9159c5ba999ee966a945f76 commit: d5fbe9b1a3d65ceeb9159c5ba999ee966a945f76 branch: master author: Pablo Aguiar committer: Pablo Galindo date: 2018-09-07T23:04:48+01:00 summary: bpo-34246: Use no mutable default args in smtplib (GH-8554) Some methods of the SMTP class use mutable default arguments. Specially `send_message` is affected as it mutates one of the args by appending items to it, which has side effects on further calls. files: A Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst M Doc/library/smtplib.rst M Lib/smtplib.py M Lib/test/test_smtplib.py M Misc/ACKS diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index e5effd0306a4..805217252ae8 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -419,7 +419,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPException`. -.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) +.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=(), rcpt_options=()) Send mail. The required arguments are an :rfc:`822` from-address string, a list of :rfc:`822` to-address strings (a bare string will be treated as a list with 1 @@ -491,7 +491,7 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ - mail_options=[], rcpt_options=[]) + mail_options=(), rcpt_options=()) This is a convenience method for calling :meth:`sendmail` with the message represented by an :class:`email.message.Message` object. The arguments have diff --git a/Lib/smtplib.py b/Lib/smtplib.py index b679875fd2c5..048c6bfb0671 100755 --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -513,7 +513,7 @@ def noop(self): """SMTP 'noop' command -- doesn't do anything :>""" return self.docmd("noop") - def mail(self, sender, options=[]): + def mail(self, sender, options=()): """SMTP 'mail' command -- begins mail xfer session. This method may raise the following exceptions: @@ -534,7 +534,7 @@ def mail(self, sender, options=[]): self.putcmd("mail", "FROM:%s%s" % (quoteaddr(sender), optionlist)) return self.getreply() - def rcpt(self, recip, options=[]): + def rcpt(self, recip, options=()): """SMTP 'rcpt' command -- indicates 1 recipient for this mail.""" optionlist = '' if options and self.does_esmtp: @@ -785,8 +785,8 @@ def starttls(self, keyfile=None, certfile=None, context=None): raise SMTPResponseException(resp, reply) return (resp, reply) - def sendmail(self, from_addr, to_addrs, msg, mail_options=[], - rcpt_options=[]): + def sendmail(self, from_addr, to_addrs, msg, mail_options=(), + rcpt_options=()): """This command performs an entire mail transaction. The arguments are: @@ -890,7 +890,7 @@ def sendmail(self, from_addr, to_addrs, msg, mail_options=[], return senderrs def send_message(self, msg, from_addr=None, to_addrs=None, - mail_options=[], rcpt_options={}): + mail_options=(), rcpt_options=()): """Converts message to a bytestring and passes it to sendmail. The arguments are as for sendmail, except that msg is an @@ -958,7 +958,7 @@ def send_message(self, msg, from_addr=None, to_addrs=None, if international: g = email.generator.BytesGenerator( bytesmsg, policy=msg.policy.clone(utf8=True)) - mail_options += ['SMTPUTF8', 'BODY=8BITMIME'] + mail_options = (*mail_options, 'SMTPUTF8', 'BODY=8BITMIME') else: g = email.generator.BytesGenerator(bytesmsg) g.flatten(msg_copy, linesep='\r\n') diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py index 495764f9aca9..8a29e98a4f30 100644 --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -20,6 +20,7 @@ import unittest from test import support, mock_socket from test.support import HOST, HOSTv4, HOSTv6 +from unittest.mock import Mock if sys.platform == 'darwin': @@ -578,6 +579,33 @@ def testNonnumericPort(self): "localhost:bogus") +class DefaultArgumentsTests(unittest.TestCase): + + def setUp(self): + self.msg = EmailMessage() + self.msg['From'] = 'P?olo ' + self.smtp = smtplib.SMTP() + self.smtp.ehlo = Mock(return_value=(200, 'OK')) + self.smtp.has_extn, self.smtp.sendmail = Mock(), Mock() + + def testSendMessage(self): + expected_mail_options = ('SMTPUTF8', 'BODY=8BITMIME') + self.smtp.send_message(self.msg) + self.smtp.send_message(self.msg) + self.assertEqual(self.smtp.sendmail.call_args_list[0][0][3], + expected_mail_options) + self.assertEqual(self.smtp.sendmail.call_args_list[1][0][3], + expected_mail_options) + + def testSendMessageWithMailOptions(self): + mail_options = ['STARTTLS'] + expected_mail_options = ('STARTTLS', 'SMTPUTF8', 'BODY=8BITMIME') + self.smtp.send_message(self.msg, None, None, mail_options) + self.assertEqual(mail_options, ['STARTTLS']) + self.assertEqual(self.smtp.sendmail.call_args_list[0][0][3], + expected_mail_options) + + # test response of client to a non-successful HELO message class BadHELOServerTests(unittest.TestCase): diff --git a/Misc/ACKS b/Misc/ACKS index 82fbc921feaa..75047d89010e 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -22,6 +22,7 @@ Eitan Adler Anton Afanasyev Ali Afshar Nitika Agarwal +Pablo S. Blum de Aguiar Jim Ahlstrom Farhan Ahmad Matthew Ahrens diff --git a/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst b/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst new file mode 100644 index 000000000000..50c91ece07ef --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst @@ -0,0 +1,2 @@ +:meth:`smtplib.SMTP.send_message` no longer modifies the content of the +*mail_options* argument. Patch by Pablo S. Blum de Aguiar. From webhook-mailer at python.org Fri Sep 7 18:10:32 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Fri, 07 Sep 2018 22:10:32 -0000 Subject: [Python-checkins] [3.6] bpo-34007: Skip traceback tests if the Program Counter is not available. (GH-9022) Message-ID: https://github.com/python/cpython/commit/963bcc8b71e4ab8c9ee9a91ed1300b6e39219821 commit: 963bcc8b71e4ab8c9ee9a91ed1300b6e39219821 branch: 3.6 author: Pablo Galindo committer: GitHub date: 2018-09-07T23:10:28+01:00 summary: [3.6] bpo-34007: Skip traceback tests if the Program Counter is not available. (GH-9022) Sometimes some versions of the shared libraries that are part of the traceback are compiled in optimised mode and the Program Counter (PC) is not present, not allowing gdb to walk the frames back. When this happens, the Python bindings of gdb raise an exception, making the test impossible to succeed. (cherry picked from commit f2ef51f8bec525b21e52988880c8a029642795ed) Co-authored-by: Pablo Galindo files: M Lib/test/test_gdb.py diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index b7554d698c9d..bedec1fb493a 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -212,6 +212,15 @@ def get_stack_trace(self, source=None, script=None, for line in errlines: if not line: continue + # bpo34007: Sometimes some versions of the shared libraries that + # are part of the traceback are compiled in optimised mode and the + # Program Counter (PC) is not present, not allowing gdb to walk the + # frames back. When this happens, the Python bindings of gdb raise + # an exception, making the test impossible to succeed. + if "PC not saved" in line: + raise unittest.SkipTest("gdb cannot walk the frame object" + " because the Program Counter is" + " not present") if not line.startswith(ignore_patterns): unexpected_errlines.append(line) From webhook-mailer at python.org Fri Sep 7 18:15:33 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 07 Sep 2018 22:15:33 -0000 Subject: [Python-checkins] bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) Message-ID: https://github.com/python/cpython/commit/73994077250bd70385cb8e7a92f24874129369d1 commit: 73994077250bd70385cb8e7a92f24874129369d1 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-07T15:15:30-07:00 summary: bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) (cherry picked from commit 4e519377b1b84c9414a360961276993d24198825) Co-authored-by: Zackery Spytz files: M PC/_msi.c diff --git a/PC/_msi.c b/PC/_msi.c index 000d81f139f3..024b2d3c9fd3 100644 --- a/PC/_msi.c +++ b/PC/_msi.c @@ -330,6 +330,10 @@ msierror(int status) code = MsiRecordGetInteger(err, 1); /* XXX code */ if (MsiFormatRecord(0, err, res, &size) == ERROR_MORE_DATA) { res = malloc(size+1); + if (res == NULL) { + MsiCloseHandle(err); + return PyErr_NoMemory(); + } MsiFormatRecord(0, err, res, &size); res[size]='\0'; } @@ -560,6 +564,9 @@ summary_getproperty(msiobj* si, PyObject *args) &fval, sval, &ssize); if (status == ERROR_MORE_DATA) { sval = malloc(ssize); + if (sval == NULL) { + return PyErr_NoMemory(); + } status = MsiSummaryInfoGetProperty(si->h, field, &type, &ival, &fval, sval, &ssize); } From webhook-mailer at python.org Fri Sep 7 18:29:34 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 07 Sep 2018 22:29:34 -0000 Subject: [Python-checkins] bpo-34246: Use no mutable default args in smtplib (GH-8554) Message-ID: https://github.com/python/cpython/commit/9835696ec4c57a9a30f1c11cfb4c5d3e121bf97c commit: 9835696ec4c57a9a30f1c11cfb4c5d3e121bf97c branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-07T15:29:27-07:00 summary: bpo-34246: Use no mutable default args in smtplib (GH-8554) Some methods of the SMTP class use mutable default arguments. Specially `send_message` is affected as it mutates one of the args by appending items to it, which has side effects on further calls. (cherry picked from commit d5fbe9b1a3d65ceeb9159c5ba999ee966a945f76) Co-authored-by: Pablo Aguiar files: A Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst M Doc/library/smtplib.rst M Lib/smtplib.py M Lib/test/test_smtplib.py M Misc/ACKS diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index e5effd0306a4..805217252ae8 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -419,7 +419,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPException`. -.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) +.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=(), rcpt_options=()) Send mail. The required arguments are an :rfc:`822` from-address string, a list of :rfc:`822` to-address strings (a bare string will be treated as a list with 1 @@ -491,7 +491,7 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ - mail_options=[], rcpt_options=[]) + mail_options=(), rcpt_options=()) This is a convenience method for calling :meth:`sendmail` with the message represented by an :class:`email.message.Message` object. The arguments have diff --git a/Lib/smtplib.py b/Lib/smtplib.py index b679875fd2c5..048c6bfb0671 100755 --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -513,7 +513,7 @@ def noop(self): """SMTP 'noop' command -- doesn't do anything :>""" return self.docmd("noop") - def mail(self, sender, options=[]): + def mail(self, sender, options=()): """SMTP 'mail' command -- begins mail xfer session. This method may raise the following exceptions: @@ -534,7 +534,7 @@ def mail(self, sender, options=[]): self.putcmd("mail", "FROM:%s%s" % (quoteaddr(sender), optionlist)) return self.getreply() - def rcpt(self, recip, options=[]): + def rcpt(self, recip, options=()): """SMTP 'rcpt' command -- indicates 1 recipient for this mail.""" optionlist = '' if options and self.does_esmtp: @@ -785,8 +785,8 @@ def starttls(self, keyfile=None, certfile=None, context=None): raise SMTPResponseException(resp, reply) return (resp, reply) - def sendmail(self, from_addr, to_addrs, msg, mail_options=[], - rcpt_options=[]): + def sendmail(self, from_addr, to_addrs, msg, mail_options=(), + rcpt_options=()): """This command performs an entire mail transaction. The arguments are: @@ -890,7 +890,7 @@ def sendmail(self, from_addr, to_addrs, msg, mail_options=[], return senderrs def send_message(self, msg, from_addr=None, to_addrs=None, - mail_options=[], rcpt_options={}): + mail_options=(), rcpt_options=()): """Converts message to a bytestring and passes it to sendmail. The arguments are as for sendmail, except that msg is an @@ -958,7 +958,7 @@ def send_message(self, msg, from_addr=None, to_addrs=None, if international: g = email.generator.BytesGenerator( bytesmsg, policy=msg.policy.clone(utf8=True)) - mail_options += ['SMTPUTF8', 'BODY=8BITMIME'] + mail_options = (*mail_options, 'SMTPUTF8', 'BODY=8BITMIME') else: g = email.generator.BytesGenerator(bytesmsg) g.flatten(msg_copy, linesep='\r\n') diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py index 495764f9aca9..8a29e98a4f30 100644 --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -20,6 +20,7 @@ import unittest from test import support, mock_socket from test.support import HOST, HOSTv4, HOSTv6 +from unittest.mock import Mock if sys.platform == 'darwin': @@ -578,6 +579,33 @@ def testNonnumericPort(self): "localhost:bogus") +class DefaultArgumentsTests(unittest.TestCase): + + def setUp(self): + self.msg = EmailMessage() + self.msg['From'] = 'P?olo ' + self.smtp = smtplib.SMTP() + self.smtp.ehlo = Mock(return_value=(200, 'OK')) + self.smtp.has_extn, self.smtp.sendmail = Mock(), Mock() + + def testSendMessage(self): + expected_mail_options = ('SMTPUTF8', 'BODY=8BITMIME') + self.smtp.send_message(self.msg) + self.smtp.send_message(self.msg) + self.assertEqual(self.smtp.sendmail.call_args_list[0][0][3], + expected_mail_options) + self.assertEqual(self.smtp.sendmail.call_args_list[1][0][3], + expected_mail_options) + + def testSendMessageWithMailOptions(self): + mail_options = ['STARTTLS'] + expected_mail_options = ('STARTTLS', 'SMTPUTF8', 'BODY=8BITMIME') + self.smtp.send_message(self.msg, None, None, mail_options) + self.assertEqual(mail_options, ['STARTTLS']) + self.assertEqual(self.smtp.sendmail.call_args_list[0][0][3], + expected_mail_options) + + # test response of client to a non-successful HELO message class BadHELOServerTests(unittest.TestCase): diff --git a/Misc/ACKS b/Misc/ACKS index bcf5604b6e21..1c53672e1cf0 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -22,6 +22,7 @@ Eitan Adler Anton Afanasyev Ali Afshar Nitika Agarwal +Pablo S. Blum de Aguiar Jim Ahlstrom Farhan Ahmad Matthew Ahrens diff --git a/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst b/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst new file mode 100644 index 000000000000..50c91ece07ef --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst @@ -0,0 +1,2 @@ +:meth:`smtplib.SMTP.send_message` no longer modifies the content of the +*mail_options* argument. Patch by Pablo S. Blum de Aguiar. From webhook-mailer at python.org Fri Sep 7 19:15:25 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Fri, 07 Sep 2018 23:15:25 -0000 Subject: [Python-checkins] bpo-34246: Make sure test_smtplib always cleans resources when finished (GH-9108) Message-ID: https://github.com/python/cpython/commit/5b7a2cb5caeb7df68e637f45a98632cbc84a51bf commit: 5b7a2cb5caeb7df68e637f45a98632cbc84a51bf branch: master author: Pablo Galindo committer: GitHub date: 2018-09-08T00:15:22+01:00 summary: bpo-34246: Make sure test_smtplib always cleans resources when finished (GH-9108) * Make sure that when some of the tests in test_smtplib fail, the allocated threads and sockets are not leaked. * Use support.join_thread() instead of thread.join() to avoid infinite blocks. files: M Lib/test/test_smtplib.py diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py index 8a29e98a4f30..0c863ed7e203 100644 --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -20,6 +20,7 @@ import unittest from test import support, mock_socket from test.support import HOST, HOSTv4, HOSTv6 +from test.support import threading_setup, threading_cleanup, join_thread from unittest.mock import Mock @@ -193,6 +194,7 @@ class DebuggingServerTests(unittest.TestCase): maxDiff = None def setUp(self): + self.thread_key = threading_setup() self.real_getfqdn = socket.getfqdn socket.getfqdn = mock_socket.getfqdn # temporarily replace sys.stdout to capture DebuggingServer output @@ -224,12 +226,15 @@ def tearDown(self): self.client_evt.set() # wait for the server thread to terminate self.serv_evt.wait() - self.thread.join() + join_thread(self.thread) # restore sys.stdout sys.stdout = self.old_stdout # restore DEBUGSTREAM smtpd.DEBUGSTREAM.close() smtpd.DEBUGSTREAM = self.old_DEBUGSTREAM + del self.thread + self.doCleanups() + threading_cleanup(*self.thread_key) def get_output_without_xpeer(self): test_output = self.output.getvalue() @@ -247,6 +252,7 @@ def testSourceAddress(self): try: smtp = smtplib.SMTP(self.host, self.port, local_hostname='localhost', timeout=3, source_address=(self.host, src_port)) + self.addCleanup(smtp.close) self.assertEqual(smtp.source_address, (self.host, src_port)) self.assertEqual(smtp.local_hostname, 'localhost') smtp.quit() @@ -257,12 +263,14 @@ def testSourceAddress(self): def testNOOP(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) expected = (250, b'OK') self.assertEqual(smtp.noop(), expected) smtp.quit() def testRSET(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) expected = (250, b'OK') self.assertEqual(smtp.rset(), expected) smtp.quit() @@ -270,6 +278,7 @@ def testRSET(self): def testELHO(self): # EHLO isn't implemented in DebuggingServer smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) expected = (250, b'\nSIZE 33554432\nHELP') self.assertEqual(smtp.ehlo(), expected) smtp.quit() @@ -277,6 +286,7 @@ def testELHO(self): def testEXPNNotImplemented(self): # EXPN isn't implemented in DebuggingServer smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) expected = (502, b'EXPN not implemented') smtp.putcmd('EXPN') self.assertEqual(smtp.getreply(), expected) @@ -284,6 +294,7 @@ def testEXPNNotImplemented(self): def testVRFY(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) expected = (252, b'Cannot VRFY user, but will accept message ' + \ b'and attempt delivery') self.assertEqual(smtp.vrfy('nobody at nowhere.com'), expected) @@ -294,6 +305,7 @@ def testSecondHELO(self): # check that a second HELO returns a message that it's a duplicate # (this behavior is specific to smtpd.SMTPChannel) smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.helo() expected = (503, b'Duplicate HELO/EHLO') self.assertEqual(smtp.helo(), expected) @@ -301,6 +313,7 @@ def testSecondHELO(self): def testHELP(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) self.assertEqual(smtp.help(), b'Supported commands: EHLO HELO MAIL ' + \ b'RCPT DATA RSET NOOP QUIT VRFY') smtp.quit() @@ -309,6 +322,7 @@ def testSend(self): # connect and send mail m = 'A test message' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.sendmail('John', 'Sally', m) # XXX(nnorwitz): this test is flaky and dies with a bad file descriptor # in asyncore. This sleep might help, but should really be fixed @@ -325,6 +339,7 @@ def testSend(self): def testSendBinary(self): m = b'A test message' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.sendmail('John', 'Sally', m) # XXX (see comment in testSend) time.sleep(0.01) @@ -340,6 +355,7 @@ def testSendNeedingDotQuote(self): # Issue 12283 m = '.A test\n.mes.sage.' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.sendmail('John', 'Sally', m) # XXX (see comment in testSend) time.sleep(0.01) @@ -354,6 +370,7 @@ def testSendNeedingDotQuote(self): def testSendNullSender(self): m = 'A test message' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.sendmail('<>', 'Sally', m) # XXX (see comment in testSend) time.sleep(0.01) @@ -371,6 +388,7 @@ def testSendNullSender(self): def testSendMessage(self): m = email.mime.text.MIMEText('A test message') smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.send_message(m, from_addr='John', to_addrs='Sally') # XXX (see comment in testSend) time.sleep(0.01) @@ -395,6 +413,7 @@ def testSendMessageWithAddresses(self): m['CC'] = 'Sally, Fred' m['Bcc'] = 'John Root , "Dinsdale" ' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.send_message(m) # XXX (see comment in testSend) time.sleep(0.01) @@ -428,6 +447,7 @@ def testSendMessageWithSomeAddresses(self): m['From'] = 'foo at bar.com' m['To'] = 'John, Dinsdale' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.send_message(m) # XXX (see comment in testSend) time.sleep(0.01) @@ -455,6 +475,7 @@ def testSendMessageWithSpecifiedAddresses(self): m['From'] = 'foo at bar.com' m['To'] = 'John, Dinsdale' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.send_message(m, from_addr='joe at example.com', to_addrs='foo at example.net') # XXX (see comment in testSend) time.sleep(0.01) @@ -485,6 +506,7 @@ def testSendMessageWithMultipleFrom(self): m['Sender'] = 'the_rescuers at Rescue-Aid-Society.com' m['To'] = 'John, Dinsdale' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.send_message(m) # XXX (see comment in testSend) time.sleep(0.01) @@ -517,6 +539,7 @@ def testSendMessageResent(self): m['Resent-To'] = 'Martha , Jeff' m['Resent-Bcc'] = 'doe at losthope.net' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) smtp.send_message(m) # XXX (see comment in testSend) time.sleep(0.01) @@ -555,6 +578,7 @@ def testSendMessageMultipleResentRaises(self): m['Resent-To'] = 'holy at grail.net' m['Resent-From'] = 'Martha , Jeff' smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + self.addCleanup(smtp.close) with self.assertRaises(ValueError): smtp.send_message(m) smtp.close() @@ -630,6 +654,7 @@ class TooLongLineTests(unittest.TestCase): respdata = b'250 OK' + (b'.' * smtplib._MAXLINE * 2) + b'\n' def setUp(self): + self.thread_key = threading_setup() self.old_stdout = sys.stdout self.output = io.StringIO() sys.stdout = self.output @@ -639,15 +664,18 @@ def setUp(self): self.sock.settimeout(15) self.port = support.bind_port(self.sock) servargs = (self.evt, self.respdata, self.sock) - thread = threading.Thread(target=server, args=servargs) - thread.start() - self.addCleanup(thread.join) + self.thread = threading.Thread(target=server, args=servargs) + self.thread.start() self.evt.wait() self.evt.clear() def tearDown(self): self.evt.wait() sys.stdout = self.old_stdout + join_thread(self.thread) + del self.thread + self.doCleanups() + threading_cleanup(*self.thread_key) def testLineTooLong(self): self.assertRaises(smtplib.SMTPResponseException, smtplib.SMTP, @@ -877,6 +905,7 @@ def handle_error(self): class SMTPSimTests(unittest.TestCase): def setUp(self): + self.thread_key = threading_setup() self.real_getfqdn = socket.getfqdn socket.getfqdn = mock_socket.getfqdn self.serv_evt = threading.Event() @@ -899,7 +928,10 @@ def tearDown(self): self.client_evt.set() # wait for the server thread to terminate self.serv_evt.wait() - self.thread.join() + join_thread(self.thread) + del self.thread + self.doCleanups() + threading_cleanup(*self.thread_key) def testBasic(self): # smoke test @@ -1162,6 +1194,7 @@ class SMTPUTF8SimTests(unittest.TestCase): maxDiff = None def setUp(self): + self.thread_key = threading_setup() self.real_getfqdn = socket.getfqdn socket.getfqdn = mock_socket.getfqdn self.serv_evt = threading.Event() @@ -1186,7 +1219,10 @@ def tearDown(self): self.client_evt.set() # wait for the server thread to terminate self.serv_evt.wait() - self.thread.join() + join_thread(self.thread) + del self.thread + self.doCleanups() + threading_cleanup(*self.thread_key) def test_test_server_supports_extensions(self): smtp = smtplib.SMTP( @@ -1283,6 +1319,7 @@ class SimSMTPAUTHInitialResponseServer(SimSMTPServer): class SMTPAUTHInitialResponseSimTests(unittest.TestCase): def setUp(self): + self.thread_key = threading_setup() self.real_getfqdn = socket.getfqdn socket.getfqdn = mock_socket.getfqdn self.serv_evt = threading.Event() @@ -1306,7 +1343,10 @@ def tearDown(self): self.client_evt.set() # wait for the server thread to terminate self.serv_evt.wait() - self.thread.join() + join_thread(self.thread) + del self.thread + self.doCleanups() + threading_cleanup(*self.thread_key) def testAUTH_PLAIN_initial_response_login(self): self.serv.add_feature('AUTH PLAIN') From webhook-mailer at python.org Fri Sep 7 19:16:20 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Fri, 07 Sep 2018 23:16:20 -0000 Subject: [Python-checkins] bpo-33083: Update "What's new" with math.factorial changes (GH-9109) Message-ID: https://github.com/python/cpython/commit/fa221d804f1bc07d992f820069bad24f176ed66d commit: fa221d804f1bc07d992f820069bad24f176ed66d branch: master author: Pablo Galindo committer: GitHub date: 2018-09-08T00:16:17+01:00 summary: bpo-33083: Update "What's new" with math.factorial changes (GH-9109) * Add elimination of non-int-like parameters in math.factorial to "What's new". files: M Doc/whatsnew/3.8.rst diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index d07896b71f23..2de7a50a471d 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -272,6 +272,9 @@ Changes in the Python API success; an exception was raised on error under Unix. (Contributed by Berker Peksag in :issue:`2122`.) +* The function :func:`math.factorial` no longer accepts arguments that are not + int-like. (Contributed by Pablo Galindo in :issue:`33083`.) + CPython bytecode changes ------------------------ From webhook-mailer at python.org Fri Sep 7 21:20:31 2018 From: webhook-mailer at python.org (Pablo Galindo) Date: Sat, 08 Sep 2018 01:20:31 -0000 Subject: [Python-checkins] [3.6] bpo-34246: Use no mutable default args in smtplib (GH-8554) (#9112) Message-ID: https://github.com/python/cpython/commit/eb6ab73f93c8b883a8d75a83560e2b4c59170d95 commit: eb6ab73f93c8b883a8d75a83560e2b4c59170d95 branch: 3.6 author: Pablo Galindo committer: GitHub date: 2018-09-08T02:20:27+01:00 summary: [3.6] bpo-34246: Use no mutable default args in smtplib (GH-8554) (#9112) Some methods of the SMTP class use mutable default arguments. Specially `send_message` is affected as it mutates one of the args by appending items to it, which has side effects on further calls.. (cherry picked from commit d5fbe9b1a3d65ceeb9159c5ba999ee966a945f76) Co-authored-by: Pablo Aguiar files: A Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst M Doc/library/smtplib.rst M Lib/smtplib.py M Lib/test/test_smtplib.py M Misc/ACKS diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index e5effd0306a4..805217252ae8 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -419,7 +419,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPException`. -.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) +.. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=(), rcpt_options=()) Send mail. The required arguments are an :rfc:`822` from-address string, a list of :rfc:`822` to-address strings (a bare string will be treated as a list with 1 @@ -491,7 +491,7 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ - mail_options=[], rcpt_options=[]) + mail_options=(), rcpt_options=()) This is a convenience method for calling :meth:`sendmail` with the message represented by an :class:`email.message.Message` object. The arguments have diff --git a/Lib/smtplib.py b/Lib/smtplib.py index b679875fd2c5..048c6bfb0671 100755 --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -513,7 +513,7 @@ def noop(self): """SMTP 'noop' command -- doesn't do anything :>""" return self.docmd("noop") - def mail(self, sender, options=[]): + def mail(self, sender, options=()): """SMTP 'mail' command -- begins mail xfer session. This method may raise the following exceptions: @@ -534,7 +534,7 @@ def mail(self, sender, options=[]): self.putcmd("mail", "FROM:%s%s" % (quoteaddr(sender), optionlist)) return self.getreply() - def rcpt(self, recip, options=[]): + def rcpt(self, recip, options=()): """SMTP 'rcpt' command -- indicates 1 recipient for this mail.""" optionlist = '' if options and self.does_esmtp: @@ -785,8 +785,8 @@ def starttls(self, keyfile=None, certfile=None, context=None): raise SMTPResponseException(resp, reply) return (resp, reply) - def sendmail(self, from_addr, to_addrs, msg, mail_options=[], - rcpt_options=[]): + def sendmail(self, from_addr, to_addrs, msg, mail_options=(), + rcpt_options=()): """This command performs an entire mail transaction. The arguments are: @@ -890,7 +890,7 @@ def sendmail(self, from_addr, to_addrs, msg, mail_options=[], return senderrs def send_message(self, msg, from_addr=None, to_addrs=None, - mail_options=[], rcpt_options={}): + mail_options=(), rcpt_options=()): """Converts message to a bytestring and passes it to sendmail. The arguments are as for sendmail, except that msg is an @@ -958,7 +958,7 @@ def send_message(self, msg, from_addr=None, to_addrs=None, if international: g = email.generator.BytesGenerator( bytesmsg, policy=msg.policy.clone(utf8=True)) - mail_options += ['SMTPUTF8', 'BODY=8BITMIME'] + mail_options = (*mail_options, 'SMTPUTF8', 'BODY=8BITMIME') else: g = email.generator.BytesGenerator(bytesmsg) g.flatten(msg_copy, linesep='\r\n') diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py index 79b3bd436876..18110191dc3b 100644 --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -18,6 +18,11 @@ import unittest from test import support, mock_socket +from unittest.mock import Mock + +HOST = "localhost" +HOSTv4 = "127.0.0.1" +HOSTv6 = "::1" try: import threading @@ -569,6 +574,33 @@ def testNonnumericPort(self): "localhost:bogus") +class DefaultArgumentsTests(unittest.TestCase): + + def setUp(self): + self.msg = EmailMessage() + self.msg['From'] = 'P?olo ' + self.smtp = smtplib.SMTP() + self.smtp.ehlo = Mock(return_value=(200, 'OK')) + self.smtp.has_extn, self.smtp.sendmail = Mock(), Mock() + + def testSendMessage(self): + expected_mail_options = ('SMTPUTF8', 'BODY=8BITMIME') + self.smtp.send_message(self.msg) + self.smtp.send_message(self.msg) + self.assertEqual(self.smtp.sendmail.call_args_list[0][0][3], + expected_mail_options) + self.assertEqual(self.smtp.sendmail.call_args_list[1][0][3], + expected_mail_options) + + def testSendMessageWithMailOptions(self): + mail_options = ['STARTTLS'] + expected_mail_options = ('STARTTLS', 'SMTPUTF8', 'BODY=8BITMIME') + self.smtp.send_message(self.msg, None, None, mail_options) + self.assertEqual(mail_options, ['STARTTLS']) + self.assertEqual(self.smtp.sendmail.call_args_list[0][0][3], + expected_mail_options) + + # test response of client to a non-successful HELO message @unittest.skipUnless(threading, 'Threading required for this test.') class BadHELOServerTests(unittest.TestCase): diff --git a/Misc/ACKS b/Misc/ACKS index 64a007236b65..bd8a5fb81f0f 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -21,6 +21,7 @@ Ron Adam Anton Afanasyev Ali Afshar Nitika Agarwal +Pablo S. Blum de Aguiar Jim Ahlstrom Farhan Ahmad Matthew Ahrens diff --git a/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst b/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst new file mode 100644 index 000000000000..50c91ece07ef --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-07-29-15-25-15.bpo-34246.xiKq-Q.rst @@ -0,0 +1,2 @@ +:meth:`smtplib.SMTP.send_message` no longer modifies the content of the +*mail_options* argument. Patch by Pablo S. Blum de Aguiar. From solipsis at pitrou.net Sat Sep 8 05:08:44 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 08 Sep 2018 09:08:44 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=7 Message-ID: <20180908090844.1.110F749308E071C7@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 3] memory blocks, sum=3 test_functools leaked [0, 3, 1] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogM7g8dC', '--timeout', '7200'] From webhook-mailer at python.org Sat Sep 8 07:48:22 2018 From: webhook-mailer at python.org (Serhiy Storchaka) Date: Sat, 08 Sep 2018 11:48:22 -0000 Subject: [Python-checkins] bpo-20104: Change the file_actions parameter of os.posix_spawn(). (GH-6725) Message-ID: https://github.com/python/cpython/commit/d700f97b627989d41cd4629dc02969f9a6b56d2f commit: d700f97b627989d41cd4629dc02969f9a6b56d2f branch: master author: Serhiy Storchaka committer: GitHub date: 2018-09-08T14:48:18+03:00 summary: bpo-20104: Change the file_actions parameter of os.posix_spawn(). (GH-6725) * Make its default value an empty tuple instead of None. * Make it a keyword-only parameter. files: M Doc/library/os.rst M Lib/test/test_posix.py M Modules/clinic/posixmodule.c.h M Modules/posixmodule.c diff --git a/Doc/library/os.rst b/Doc/library/os.rst index b8d6fffb303b..bc8d5a8abbf3 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3394,7 +3394,7 @@ written in Python, such as a mail server's external command delivery program. subprocesses. -.. function:: posix_spawn(path, argv, env, file_actions=None, /, *, \ +.. function:: posix_spawn(path, argv, env, *, file_actions=None, \ setpgroup=None, resetids=False, setsigmask=(), \ setsigdef=(), scheduler=None) @@ -3402,7 +3402,8 @@ written in Python, such as a mail server's external command delivery program. Most users should use :func:`subprocess.run` instead of :func:`posix_spawn`. - The *path*, *args*, and *env* arguments are similar to :func:`execve`. + The positional-only arguments *path*, *args*, and *env* are similar to + :func:`execve`. The *file_actions* argument may be a sequence of tuples describing actions to take on specific file descriptors in the child process between the C diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index dafe9c1350e4..7a2fc263cb27 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1499,8 +1499,7 @@ def test_returns_pid(self): pidfile.write(str(os.getpid())) """ args = self.python_args('-c', script) - pid = posix.posix_spawn(args[0], args, - os.environ) + pid = posix.posix_spawn(args[0], args, os.environ) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) with open(pidfile) as f: self.assertEqual(f.read(), str(pid)) @@ -1538,7 +1537,7 @@ def test_empty_file_actions(self): self.NOOP_PROGRAM[0], self.NOOP_PROGRAM, os.environ, - [] + file_actions=[] ) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) @@ -1691,37 +1690,38 @@ def test_multiple_file_actions(self): ] pid = posix.posix_spawn(self.NOOP_PROGRAM[0], self.NOOP_PROGRAM, - os.environ, file_actions) + os.environ, + file_actions=file_actions) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) def test_bad_file_actions(self): args = self.NOOP_PROGRAM with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [None]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[None]) with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [()]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[()]) with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [(None,)]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[(None,)]) with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [(12345,)]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[(12345,)]) with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [(os.POSIX_SPAWN_CLOSE,)]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[(os.POSIX_SPAWN_CLOSE,)]) with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [(os.POSIX_SPAWN_CLOSE, 1, 2)]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[(os.POSIX_SPAWN_CLOSE, 1, 2)]) with self.assertRaises(TypeError): - posix.posix_spawn(args[0], args, - os.environ, [(os.POSIX_SPAWN_CLOSE, None)]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[(os.POSIX_SPAWN_CLOSE, None)]) with self.assertRaises(ValueError): - posix.posix_spawn(args[0], args, - os.environ, - [(os.POSIX_SPAWN_OPEN, 3, __file__ + '\0', - os.O_RDONLY, 0)]) + posix.posix_spawn(args[0], args, os.environ, + file_actions=[(os.POSIX_SPAWN_OPEN, + 3, __file__ + '\0', + os.O_RDONLY, 0)]) def test_open_file(self): outfile = support.TESTFN @@ -1736,8 +1736,8 @@ def test_open_file(self): stat.S_IRUSR | stat.S_IWUSR), ] args = self.python_args('-c', script) - pid = posix.posix_spawn(args[0], args, - os.environ, file_actions) + pid = posix.posix_spawn(args[0], args, os.environ, + file_actions=file_actions) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) with open(outfile) as f: self.assertEqual(f.read(), 'hello') @@ -1754,9 +1754,8 @@ def test_close_file(self): closefile.write('is closed %d' % e.errno) """ args = self.python_args('-c', script) - pid = posix.posix_spawn(args[0], args, - os.environ, - [(os.POSIX_SPAWN_CLOSE, 0),]) + pid = posix.posix_spawn(args[0], args, os.environ, + file_actions=[(os.POSIX_SPAWN_CLOSE, 0),]) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) with open(closefile) as f: self.assertEqual(f.read(), 'is closed %d' % errno.EBADF) @@ -1773,8 +1772,8 @@ def test_dup2(self): (os.POSIX_SPAWN_DUP2, childfile.fileno(), 1), ] args = self.python_args('-c', script) - pid = posix.posix_spawn(args[0], args, - os.environ, file_actions) + pid = posix.posix_spawn(args[0], args, os.environ, + file_actions=file_actions) self.assertEqual(os.waitpid(pid, 0), (pid, 0)) with open(dupfile) as f: self.assertEqual(f.read(), 'hello') diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index f7767c4af0bb..c3849a93c0c2 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -1730,7 +1730,7 @@ os_execve(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *k #if defined(HAVE_POSIX_SPAWN) PyDoc_STRVAR(os_posix_spawn__doc__, -"posix_spawn($module, path, argv, env, file_actions=None, /, *,\n" +"posix_spawn($module, path, argv, env, /, *, file_actions=(),\n" " setpgroup=None, resetids=False, setsigmask=(),\n" " setsigdef=(), scheduler=None)\n" "--\n" @@ -1769,12 +1769,12 @@ static PyObject * os_posix_spawn(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; - static const char * const _keywords[] = {"", "", "", "", "setpgroup", "resetids", "setsigmask", "setsigdef", "scheduler", NULL}; - static _PyArg_Parser _parser = {"O&OO|O$OiOOO:posix_spawn", _keywords, 0}; + static const char * const _keywords[] = {"", "", "", "file_actions", "setpgroup", "resetids", "setsigmask", "setsigdef", "scheduler", NULL}; + static _PyArg_Parser _parser = {"O&OO|$OOiOOO:posix_spawn", _keywords, 0}; path_t path = PATH_T_INITIALIZE("posix_spawn", "path", 0, 0); PyObject *argv; PyObject *env; - PyObject *file_actions = Py_None; + PyObject *file_actions = NULL; PyObject *setpgroup = NULL; int resetids = 0; PyObject *setsigmask = NULL; @@ -6648,4 +6648,4 @@ os_getrandom(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject #ifndef OS_GETRANDOM_METHODDEF #define OS_GETRANDOM_METHODDEF #endif /* !defined(OS_GETRANDOM_METHODDEF) */ -/*[clinic end generated code: output=ef78384ae88712e1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=758ee0434fb03d90 input=a9049054013a1b77]*/ diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 4a8a8d70ae42..c5d9a17b002e 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -5405,10 +5405,10 @@ os.posix_spawn Tuple or list of strings. env: object Dictionary of strings mapping to strings. - file_actions: object = None - A sequence of file action tuples. / * + file_actions: object(c_default='NULL') = () + A sequence of file action tuples. setpgroup: object = NULL The pgroup to use with the POSIX_SPAWN_SETPGROUP flag. resetids: bool(accept={int}) = False @@ -5419,6 +5419,7 @@ os.posix_spawn The sigmask to use with the POSIX_SPAWN_SETSIGDEF flag. scheduler: object = NULL A tuple with the scheduler policy (optional) and parameters. + Execute the program specified by path in a new process. [clinic start generated code]*/ @@ -5427,7 +5428,7 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, PyObject *env, PyObject *file_actions, PyObject *setpgroup, int resetids, PyObject *setsigmask, PyObject *setsigdef, PyObject *scheduler) -/*[clinic end generated code: output=45dfa4c515d09f2c input=2d7a7578430a90f0]*/ +/*[clinic end generated code: output=45dfa4c515d09f2c input=2891c2f1d457e39b]*/ { EXECV_CHAR **argvlist = NULL; EXECV_CHAR **envlist = NULL; @@ -5477,7 +5478,7 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, goto exit; } - if (file_actions != Py_None) { + if (file_actions != NULL) { /* There is a bug in old versions of glibc that makes some of the * helper functions for manipulating file actions not copy the provided * buffers. The problem is that posix_spawn_file_actions_addopen does not From webhook-mailer at python.org Sat Sep 8 16:31:31 2018 From: webhook-mailer at python.org (=?utf-8?q?=C3=89ric?= Araujo) Date: Sat, 08 Sep 2018 20:31:31 -0000 Subject: [Python-checkins] bpo-34421 avoid unicode error in distutils logging (GH-8799) Message-ID: https://github.com/python/cpython/commit/0afada163c7ef25c3a9d46ed445481fb69f2ecaf commit: 0afada163c7ef25c3a9d46ed445481fb69f2ecaf branch: master author: Julien Malard committer: ?ric Araujo date: 2018-09-08T16:31:26-04:00 summary: bpo-34421 avoid unicode error in distutils logging (GH-8799) This caused installation errors in some cases on Windows. Patch by Julien Malard. files: A Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst M Lib/distutils/log.py diff --git a/Lib/distutils/log.py b/Lib/distutils/log.py index b301a8338c20..3a6602bc8b8e 100644 --- a/Lib/distutils/log.py +++ b/Lib/distutils/log.py @@ -31,7 +31,10 @@ def _log(self, level, msg, args): # emulate backslashreplace error handler encoding = stream.encoding msg = msg.encode(encoding, "backslashreplace").decode(encoding) - stream.write('%s\n' % msg) + try: + stream.write('%s\n' % msg) + except UnicodeEncodeError: + stream.write('%s\n' % msg.encode('unicode-escape').decode('ascii')) stream.flush() def log(self, level, msg, *args): diff --git a/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst b/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst new file mode 100644 index 000000000000..cc1db086f0c4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst @@ -0,0 +1 @@ +Fix distutils logging for non-ASCII strings. This caused installation issues on Windows. From webhook-mailer at python.org Sat Sep 8 16:44:24 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 08 Sep 2018 20:44:24 -0000 Subject: [Python-checkins] bpo-34421 avoid unicode error in distutils logging (GH-8799) Message-ID: https://github.com/python/cpython/commit/3b36642924a51e6bceb7033916c3049764817166 commit: 3b36642924a51e6bceb7033916c3049764817166 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-08T13:44:20-07:00 summary: bpo-34421 avoid unicode error in distutils logging (GH-8799) This caused installation errors in some cases on Windows. Patch by Julien Malard. (cherry picked from commit 0afada163c7ef25c3a9d46ed445481fb69f2ecaf) Co-authored-by: Julien Malard files: A Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst M Lib/distutils/log.py diff --git a/Lib/distutils/log.py b/Lib/distutils/log.py index b301a8338c20..3a6602bc8b8e 100644 --- a/Lib/distutils/log.py +++ b/Lib/distutils/log.py @@ -31,7 +31,10 @@ def _log(self, level, msg, args): # emulate backslashreplace error handler encoding = stream.encoding msg = msg.encode(encoding, "backslashreplace").decode(encoding) - stream.write('%s\n' % msg) + try: + stream.write('%s\n' % msg) + except UnicodeEncodeError: + stream.write('%s\n' % msg.encode('unicode-escape').decode('ascii')) stream.flush() def log(self, level, msg, *args): diff --git a/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst b/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst new file mode 100644 index 000000000000..cc1db086f0c4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst @@ -0,0 +1 @@ +Fix distutils logging for non-ASCII strings. This caused installation issues on Windows. From webhook-mailer at python.org Sat Sep 8 16:53:02 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 08 Sep 2018 20:53:02 -0000 Subject: [Python-checkins] bpo-34421 avoid unicode error in distutils logging (GH-8799) Message-ID: https://github.com/python/cpython/commit/77b92b15a5e5c84b91d3fd9d02f63db432fa8903 commit: 77b92b15a5e5c84b91d3fd9d02f63db432fa8903 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-08T13:52:59-07:00 summary: bpo-34421 avoid unicode error in distutils logging (GH-8799) This caused installation errors in some cases on Windows. Patch by Julien Malard. (cherry picked from commit 0afada163c7ef25c3a9d46ed445481fb69f2ecaf) Co-authored-by: Julien Malard files: A Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst M Lib/distutils/log.py diff --git a/Lib/distutils/log.py b/Lib/distutils/log.py index b301a8338c20..3a6602bc8b8e 100644 --- a/Lib/distutils/log.py +++ b/Lib/distutils/log.py @@ -31,7 +31,10 @@ def _log(self, level, msg, args): # emulate backslashreplace error handler encoding = stream.encoding msg = msg.encode(encoding, "backslashreplace").decode(encoding) - stream.write('%s\n' % msg) + try: + stream.write('%s\n' % msg) + except UnicodeEncodeError: + stream.write('%s\n' % msg.encode('unicode-escape').decode('ascii')) stream.flush() def log(self, level, msg, *args): diff --git a/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst b/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst new file mode 100644 index 000000000000..cc1db086f0c4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-07-10-57-00.bpo-34421.AKJISD.rst @@ -0,0 +1 @@ +Fix distutils logging for non-ASCII strings. This caused installation issues on Windows. From solipsis at pitrou.net Sun Sep 9 05:09:51 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 09 Sep 2018 09:09:51 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=5 Message-ID: <20180909090951.1.710B374100B5B5C1@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [-7, 1, 7] memory blocks, sum=1 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_fork leaked [0, -2, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogRauWFq', '--timeout', '7200'] From webhook-mailer at python.org Sun Sep 9 07:26:58 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Sun, 09 Sep 2018 11:26:58 -0000 Subject: [Python-checkins] [3.6] bpo-34604: Fix possible mojibake in pwd.getpwnam() and grp.getgrnam() (GH-9098) (GH-9105) Message-ID: https://github.com/python/cpython/commit/1bcd891c200b8122493ddad5a203331e1a3bfcb5 commit: 1bcd891c200b8122493ddad5a203331e1a3bfcb5 branch: 3.6 author: William Grzybowski committer: Victor Stinner date: 2018-09-09T13:26:48+02:00 summary: [3.6] bpo-34604: Fix possible mojibake in pwd.getpwnam() and grp.getgrnam() (GH-9098) (GH-9105) Pass the user/group name as Unicode to the formatting function, instead of always decoding a bytes string from UTF-8.. (cherry picked from commit 28658485a54ad5f9df52ecc12d9046269f1654ec) Co-authored-by: William Grzybowski files: A Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst M Modules/grpmodule.c M Modules/pwdmodule.c diff --git a/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst b/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst new file mode 100644 index 000000000000..562a69124b3d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst @@ -0,0 +1,2 @@ +Fix possible mojibake in the error message of `pwd.getpwnam` and +`grp.getgrnam`. Patch by William Grzybowski. diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c index f577fd3ab4ec..43e45ef7aad5 100644 --- a/Modules/grpmodule.c +++ b/Modules/grpmodule.c @@ -156,7 +156,7 @@ grp_getgrnam_impl(PyObject *module, PyObject *name) goto out; if ((p = getgrnam(name_chars)) == NULL) { - PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %s", name_chars); + PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %S", name); goto out; } retval = mkgrent(p); diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c index bbef2de9c522..21c2b546f6dd 100644 --- a/Modules/pwdmodule.c +++ b/Modules/pwdmodule.c @@ -163,7 +163,7 @@ pwd_getpwnam_impl(PyObject *module, PyObject *arg) goto out; if ((p = getpwnam(name)) == NULL) { PyErr_Format(PyExc_KeyError, - "getpwnam(): name not found: %s", name); + "getpwnam(): name not found: %S", arg); goto out; } retval = mkpwent(p); From webhook-mailer at python.org Sun Sep 9 07:27:35 2018 From: webhook-mailer at python.org (Victor Stinner) Date: Sun, 09 Sep 2018 11:27:35 -0000 Subject: [Python-checkins] [3.7] bpo-34604: Fix possible mojibake in pwd.getpwnam() and grp.getgrnam() (GH-9098) (GH-9104) Message-ID: https://github.com/python/cpython/commit/7a633ed79cfba2cfc0f80410ddcaeecadc2030e9 commit: 7a633ed79cfba2cfc0f80410ddcaeecadc2030e9 branch: 3.7 author: William Grzybowski committer: Victor Stinner date: 2018-09-09T13:27:31+02:00 summary: [3.7] bpo-34604: Fix possible mojibake in pwd.getpwnam() and grp.getgrnam() (GH-9098) (GH-9104) Pass the user/group name as Unicode to the formatting function, instead of always decoding a bytes string from UTF-8.. (cherry picked from commit 28658485a54ad5f9df52ecc12d9046269f1654ec) Co-authored-by: William Grzybowski files: A Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst M Modules/grpmodule.c M Modules/pwdmodule.c diff --git a/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst b/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst new file mode 100644 index 000000000000..562a69124b3d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-07-10-16-34.bpo-34604.xL7-kG.rst @@ -0,0 +1,2 @@ +Fix possible mojibake in the error message of `pwd.getpwnam` and +`grp.getgrnam`. Patch by William Grzybowski. diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c index f577fd3ab4ec..43e45ef7aad5 100644 --- a/Modules/grpmodule.c +++ b/Modules/grpmodule.c @@ -156,7 +156,7 @@ grp_getgrnam_impl(PyObject *module, PyObject *name) goto out; if ((p = getgrnam(name_chars)) == NULL) { - PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %s", name_chars); + PyErr_Format(PyExc_KeyError, "getgrnam(): name not found: %S", name); goto out; } retval = mkgrent(p); diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c index bbef2de9c522..21c2b546f6dd 100644 --- a/Modules/pwdmodule.c +++ b/Modules/pwdmodule.c @@ -163,7 +163,7 @@ pwd_getpwnam_impl(PyObject *module, PyObject *arg) goto out; if ((p = getpwnam(name)) == NULL) { PyErr_Format(PyExc_KeyError, - "getpwnam(): name not found: %s", name); + "getpwnam(): name not found: %S", arg); goto out; } retval = mkpwent(p); From webhook-mailer at python.org Mon Sep 10 02:27:35 2018 From: webhook-mailer at python.org (Serhiy Storchaka) Date: Mon, 10 Sep 2018 06:27:35 -0000 Subject: [Python-checkins] Remove unneeded PyUnicode_READY() in tokenizer.c (GH-9114) Message-ID: https://github.com/python/cpython/commit/5061a74a4c8686764a299d82df048bf858dd263b commit: 5061a74a4c8686764a299d82df048bf858dd263b branch: master author: Zackery Spytz committer: Serhiy Storchaka date: 2018-09-10T09:27:31+03:00 summary: Remove unneeded PyUnicode_READY() in tokenizer.c (GH-9114) files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 6566fdead380..fc75bae53766 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1324,7 +1324,7 @@ verify_identifier(struct tok_state *tok) if (tok->decoding_erred) return 0; s = PyUnicode_DecodeUTF8(tok->start, tok->cur - tok->start, NULL); - if (s == NULL || PyUnicode_READY(s) == -1) { + if (s == NULL) { if (PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) { PyErr_Clear(); tok->done = E_IDENTIFIER; From webhook-mailer at python.org Mon Sep 10 05:07:27 2018 From: webhook-mailer at python.org (Xiang Zhang) Date: Mon, 10 Sep 2018 09:07:27 -0000 Subject: [Python-checkins] Revert "Fix misindented yaml in logging how to example (GH-8604)" (GH-9081) Message-ID: https://github.com/python/cpython/commit/290a60bd8af7b1d7e7931aa4dd4eace60d355d76 commit: 290a60bd8af7b1d7e7931aa4dd4eace60d355d76 branch: master author: R?my HUBSCHER committer: Xiang Zhang date: 2018-09-10T17:07:15+08:00 summary: Revert "Fix misindented yaml in logging how to example (GH-8604)" (GH-9081) This reverts commit 10b59f1b019cd00c940dd7f4a74c4f667a20f25f. files: M Doc/howto/logging.rst diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst index 6f47baeee2f5..47b5c680c424 100644 --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -750,9 +750,9 @@ the new dictionary-based approach: level: DEBUG handlers: [console] propagate: no - root: - level: DEBUG - handlers: [console] + root: + level: DEBUG + handlers: [console] For more information about logging using a dictionary, see :ref:`logging-config-api`. From solipsis at pitrou.net Mon Sep 10 05:07:33 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 10 Sep 2018 09:07:33 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=3 Message-ID: <20180910090733.1.5535083142C260FE@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_forkserver leaked [-2, 1, 0] memory blocks, sum=-1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflog8sHDRk', '--timeout', '7200'] From webhook-mailer at python.org Mon Sep 10 05:13:24 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 09:13:24 -0000 Subject: [Python-checkins] Revert "Fix misindented yaml in logging how to example (GH-8604)" (GH-9081) Message-ID: https://github.com/python/cpython/commit/9c9ac9e69735a2854c2e0d41948385110b139abe commit: 9c9ac9e69735a2854c2e0d41948385110b139abe branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T02:13:18-07:00 summary: Revert "Fix misindented yaml in logging how to example (GH-8604)" (GH-9081) This reverts commit 10b59f1b019cd00c940dd7f4a74c4f667a20f25f. (cherry picked from commit 290a60bd8af7b1d7e7931aa4dd4eace60d355d76) Co-authored-by: R?my HUBSCHER files: M Doc/howto/logging.rst diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst index 6f47baeee2f5..47b5c680c424 100644 --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -750,9 +750,9 @@ the new dictionary-based approach: level: DEBUG handlers: [console] propagate: no - root: - level: DEBUG - handlers: [console] + root: + level: DEBUG + handlers: [console] For more information about logging using a dictionary, see :ref:`logging-config-api`. From webhook-mailer at python.org Mon Sep 10 05:19:25 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 09:19:25 -0000 Subject: [Python-checkins] Revert "Fix misindented yaml in logging how to example (GH-8604)" (GH-9081) Message-ID: https://github.com/python/cpython/commit/8eda821d1b8db374bf963cf18b5c7f063366b062 commit: 8eda821d1b8db374bf963cf18b5c7f063366b062 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T02:19:20-07:00 summary: Revert "Fix misindented yaml in logging how to example (GH-8604)" (GH-9081) This reverts commit 10b59f1b019cd00c940dd7f4a74c4f667a20f25f. (cherry picked from commit 290a60bd8af7b1d7e7931aa4dd4eace60d355d76) Co-authored-by: R?my HUBSCHER files: M Doc/howto/logging.rst diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst index 78a237f5ae63..9914f8d154ab 100644 --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -750,9 +750,9 @@ the new dictionary-based approach: level: DEBUG handlers: [console] propagate: no - root: - level: DEBUG - handlers: [console] + root: + level: DEBUG + handlers: [console] For more information about logging using a dictionary, see :ref:`logging-config-api`. From webhook-mailer at python.org Mon Sep 10 05:42:14 2018 From: webhook-mailer at python.org (Xiang Zhang) Date: Mon, 10 Sep 2018 09:42:14 -0000 Subject: [Python-checkins] Test dict values iterator pickling with pickle.HIGHEST_PROTOCOL. (GH-9052) Message-ID: https://github.com/python/cpython/commit/1f36bf6077d93cb43fd84bea4a8a625fa772d1fa commit: 1f36bf6077d93cb43fd84bea4a8a625fa772d1fa branch: master author: Sergey Fedoseev committer: Xiang Zhang date: 2018-09-10T17:42:09+08:00 summary: Test dict values iterator pickling with pickle.HIGHEST_PROTOCOL. (GH-9052) files: M Lib/test/test_dict.py diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index aa149d31eb0e..38521bbf6630 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1056,7 +1056,7 @@ def test_itemiterator_pickling(self): self.assertEqual(dict(it), data) def test_valuesiterator_pickling(self): - for proto in range(pickle.HIGHEST_PROTOCOL): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): data = {1:"a", 2:"b", 3:"c"} # data.values() isn't picklable, only its iterator it = iter(data.values()) From webhook-mailer at python.org Mon Sep 10 06:32:57 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 10:32:57 -0000 Subject: [Python-checkins] Test dict values iterator pickling with pickle.HIGHEST_PROTOCOL. (GH-9052) Message-ID: https://github.com/python/cpython/commit/9f5351edbe8610f75174013be66a767fecf78fe2 commit: 9f5351edbe8610f75174013be66a767fecf78fe2 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T03:32:53-07:00 summary: Test dict values iterator pickling with pickle.HIGHEST_PROTOCOL. (GH-9052) (cherry picked from commit 1f36bf6077d93cb43fd84bea4a8a625fa772d1fa) Co-authored-by: Sergey Fedoseev files: M Lib/test/test_dict.py diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index 4386eda3ae48..639e05f74a8a 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1010,7 +1010,7 @@ def test_itemiterator_pickling(self): self.assertEqual(dict(it), data) def test_valuesiterator_pickling(self): - for proto in range(pickle.HIGHEST_PROTOCOL): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): data = {1:"a", 2:"b", 3:"c"} # data.values() isn't picklable, only its iterator it = iter(data.values()) From webhook-mailer at python.org Mon Sep 10 06:45:57 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 10:45:57 -0000 Subject: [Python-checkins] Test dict values iterator pickling with pickle.HIGHEST_PROTOCOL. (GH-9052) Message-ID: https://github.com/python/cpython/commit/e9119a5de67d5525f3be957fda239143453513bd commit: e9119a5de67d5525f3be957fda239143453513bd branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T03:45:54-07:00 summary: Test dict values iterator pickling with pickle.HIGHEST_PROTOCOL. (GH-9052) (cherry picked from commit 1f36bf6077d93cb43fd84bea4a8a625fa772d1fa) Co-authored-by: Sergey Fedoseev files: M Lib/test/test_dict.py diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index aa149d31eb0e..38521bbf6630 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1056,7 +1056,7 @@ def test_itemiterator_pickling(self): self.assertEqual(dict(it), data) def test_valuesiterator_pickling(self): - for proto in range(pickle.HIGHEST_PROTOCOL): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): data = {1:"a", 2:"b", 3:"c"} # data.values() isn't picklable, only its iterator it = iter(data.values()) From webhook-mailer at python.org Mon Sep 10 09:11:08 2018 From: webhook-mailer at python.org (Tal Einat) Date: Mon, 10 Sep 2018 13:11:08 -0000 Subject: [Python-checkins] bpo-30977: rework code changes according to post-merge code review (GH-9106) Message-ID: https://github.com/python/cpython/commit/54752533b2ed1c898ffe5ec2e795c6910ee46a39 commit: 54752533b2ed1c898ffe5ec2e795c6910ee46a39 branch: master author: Tal Einat committer: GitHub date: 2018-09-10T16:11:04+03:00 summary: bpo-30977: rework code changes according to post-merge code review (GH-9106) also mention the change and its consequences in What's New files: M Doc/whatsnew/3.8.rst M Lib/test/test_uuid.py M Lib/uuid.py diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index 2de7a50a471d..a9b689d53728 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -131,6 +131,10 @@ Optimizations objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes. (Contributed by Inada Naoki in :issue:`33597`) +* :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint. + Note that this means that instances can no longer be weak-referenced and + that arbitrary attributes can no longer be added to them. + Build and C API Changes ======================= @@ -275,6 +279,9 @@ Changes in the Python API * The function :func:`math.factorial` no longer accepts arguments that are not int-like. (Contributed by Pablo Galindo in :issue:`33083`.) +* :class:`uuid.UUID` now uses ``__slots__``, therefore instances can no longer + be weak-referenced and attributes can no longer be added. + CPython bytecode changes ------------------------ diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py index 9ec59d519573..c1b35c4a7f6c 100644 --- a/Lib/test/test_uuid.py +++ b/Lib/test/test_uuid.py @@ -2,6 +2,8 @@ from test import support import builtins import contextlib +import copy +from functools import partial import io import os import pickle @@ -313,63 +315,139 @@ def test_getnode(self): node2 = self.uuid.getnode() self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2)) - def _setup_for_pickle(self): - orig_uuid = sys.modules.get('uuid') - sys.modules['uuid'] = self.uuid - - def restore_uuid_module(): - if orig_uuid is not None: - sys.modules['uuid'] = orig_uuid - else: - del sys.modules['uuid'] - self.addCleanup(restore_uuid_module) - def test_pickle_roundtrip(self): - self._setup_for_pickle() - - u = self.uuid.UUID('12345678123456781234567812345678') - self.assertEqual(u, pickle.loads(pickle.dumps(u))) + def check(actual, expected): + self.assertEqual(actual, expected) + self.assertEqual(actual.is_safe, expected.is_safe) + + with support.swap_item(sys.modules, 'uuid', self.uuid): + for is_safe in self.uuid.SafeUUID: + u = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5', + is_safe=is_safe) + check(copy.copy(u), u) + check(copy.deepcopy(u), u) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(protocol=proto): + check(pickle.loads(pickle.dumps(u, proto)), u) def test_unpickle_previous_python_versions(self): - self._setup_for_pickle() - - u = self.uuid.UUID('12345678123456781234567812345678') - - # Python 2.7 protocol 0-2 pickles of u - py27_pickles = [ - b'ccopy_reg\n_reconstructor\np0\n(cuuid\nUUID\np1\nc__builtin__\nob' - b'ject\np2\nNtp3\nRp4\n(dp5\nS\'int\'\np6\nL24197857161011715162171' - b'839636988778104L\nsb.', - b'ccopy_reg\n_reconstructor\nq\x00(cuuid\nUUID\nq\x01c__builtin__\n' - b'object\nq\x02Ntq\x03Rq\x04}q\x05U\x03intq\x06L2419785716101171516' - b'2171839636988778104L\nsb.', - b'\x80\x02cuuid\nUUID\nq\x00)\x81q\x01}q\x02U\x03intq\x03\x8a\x10xV' - b'4\x12xV4\x12xV4\x12xV4\x12sb.', + def check(actual, expected): + self.assertEqual(actual, expected) + self.assertEqual(actual.is_safe, expected.is_safe) + + pickled_uuids = [ + # Python 2.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dS\'int\'\nL287307832597519156748809049798316161701L\nsb.', + # Python 2.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}U\x03intL287307832597519156748809049798316161701L\nsb.', + # Python 2.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}U\x03int\x8a\x11\xa5z\xecz\nI\xdf}' + b'\xde\xa0Bf\xcey%\xd8\x00sb.', + # Python 3.6, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsb.', + # Python 3.6, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}X\x03\x00\x00\x00intL287307832597519156748809049798316161701L' + b'\nsb.', + # Python 3.6, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}X\x03\x00\x00\x00int\x8a\x11\xa5z\xec' + b'z\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00sb.', + # Python 3.6, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}X\x03\x00\x00\x00int\x8a\x11\xa5z\xec' + b'z\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00sb.', + # Python 3.6, protocol 4 + b'\x80\x04\x95+\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x8c\x04UUI' + b'D\x93)\x81}\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0Bf\xcey%' + b'\xd8\x00sb.', + # Python 3.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsVis_safe\n' + b'cuuid\nSafeUUID\n(NtRsb.', + # Python 3.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}(X\x03\x00\x00\x00intL287307832597519156748809049798316161701' + b'L\nX\x07\x00\x00\x00is_safecuuid\nSafeUUID\n(NtRub.', + # Python 3.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nN\x85Rub.', + # Python 3.7, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nN\x85Rub.', + # Python 3.7, protocol 4 + b'\x80\x04\x95F\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c' + b'\x04UUID\x93)\x81}(\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0' + b'Bf\xcey%\xd8\x00\x8c\x07is_safeh\x00\x8c\x08SafeUUID\x93N\x85Rub' + b'.', + ] + pickled_uuids_safe = [ + # Python 3.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsVis_safe\n' + b'cuuid\nSafeUUID\n(I0\ntRsb.', + # Python 3.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}(X\x03\x00\x00\x00intL287307832597519156748809049798316161701' + b'L\nX\x07\x00\x00\x00is_safecuuid\nSafeUUID\n(K\x00tRub.', + # Python 3.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nK\x00\x85Rub.', + # Python 3.7, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nK\x00\x85Rub.', + # Python 3.7, protocol 4 + b'\x80\x04\x95G\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c' + b'\x04UUID\x93)\x81}(\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0' + b'Bf\xcey%\xd8\x00\x8c\x07is_safeh\x00\x8c\x08SafeUUID\x93K\x00' + b'\x85Rub.', ] - # Python 3.6 protocol 0-4 pickles of u - py36_pickles = [ - b'ccopy_reg\n_reconstructor\np0\n(cuuid\nUUID\np1\nc__builtin__\nob' - b'ject\np2\nNtp3\nRp4\n(dp5\nVint\np6\nL241978571610117151621718396' - b'36988778104L\nsb.', - b'ccopy_reg\n_reconstructor\nq\x00(cuuid\nUUID\nq\x01c__builtin__\n' - b'object\nq\x02Ntq\x03Rq\x04}q\x05X\x03\x00\x00\x00intq\x06L2419785' - b'7161011715162171839636988778104L\nsb.', - b'\x80\x02cuuid\nUUID\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00intq' - b'\x03\x8a\x10xV4\x12xV4\x12xV4\x12xV4\x12sb.', - b'\x80\x03cuuid\nUUID\nq\x00)\x81q\x01}q\x02X\x03\x00\x00\x00intq' - b'\x03\x8a\x10xV4\x12xV4\x12xV4\x12xV4\x12sb.', - b'\x80\x04\x950\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c\x04' - b'UUID\x94\x93\x94)\x81\x94}\x94\x8c\x03int\x94\x8a\x10xV4\x12xV4' - b'\x12xV4\x12xV4\x12sb.', + pickled_uuids_unsafe = [ + # Python 3.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsVis_safe\n' + b'cuuid\nSafeUUID\n(I-1\ntRsb.', + # Python 3.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}(X\x03\x00\x00\x00intL287307832597519156748809049798316161701' + b'L\nX\x07\x00\x00\x00is_safecuuid\nSafeUUID\n(J\xff\xff\xff\xfftR' + b'ub.', + # Python 3.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nJ\xff\xff\xff\xff\x85Rub.', + # Python 3.7, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nJ\xff\xff\xff\xff\x85Rub.', + # Python 3.7, protocol 4 + b'\x80\x04\x95J\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c' + b'\x04UUID\x93)\x81}(\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0' + b'Bf\xcey%\xd8\x00\x8c\x07is_safeh\x00\x8c\x08SafeUUID\x93J\xff' + b'\xff\xff\xff\x85Rub.', ] - for pickled in py27_pickles + py36_pickles: - unpickled = pickle.loads(pickled) - self.assertEqual(unpickled, u) - # is_safe was added in 3.7. When unpickling values from older - # versions, is_safe will be missing, so it should be set to - # SafeUUID.unknown. - self.assertEqual(unpickled.is_safe, self.uuid.SafeUUID.unknown) + u = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5') + u_safe = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5', + is_safe=self.uuid.SafeUUID.safe) + u_unsafe = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5', + is_safe=self.uuid.SafeUUID.unsafe) + + with support.swap_item(sys.modules, 'uuid', self.uuid): + for pickled in pickled_uuids: + # is_safe was added in 3.7. When unpickling values from older + # versions, is_safe will be missing, so it should be set to + # SafeUUID.unknown. + check(pickle.loads(pickled), u) + for pickled in pickled_uuids_safe: + check(pickle.loads(pickled), u_safe) + for pickled in pickled_uuids_unsafe: + check(pickle.loads(pickled), u_unsafe) # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers # need not necessarily be 48 bits (e.g., EUI-64). diff --git a/Lib/uuid.py b/Lib/uuid.py index 515388221510..073ca711ab42 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -207,26 +207,19 @@ def __init__(self, hex=None, bytes=None, bytes_le=None, fields=None, object.__setattr__(self, 'is_safe', is_safe) def __getstate__(self): - d = {attr: getattr(self, attr) for attr in self.__slots__} - # is_safe is a SafeUUID instance. Return just its value, so that - # it can be unpickled in older Python versions without SafeUUID. - d['is_safe'] = d['is_safe'].value + d = {'int': self.int} + if self.is_safe != SafeUUID.unknown: + # is_safe is a SafeUUID instance. Return just its value, so that + # it can be un-pickled in older Python versions without SafeUUID. + d['is_safe'] = self.is_safe.value return d def __setstate__(self, state): - # is_safe was added in 3.7 - state.setdefault('is_safe', SafeUUID.unknown.value) - - for attr in self.__slots__: - value = state[attr] - - # for is_safe, restore the SafeUUID from the stored value - if attr == 'is_safe': - try: - value = SafeUUID(value) - except ValueError: - value = SafeUUID.unknown - object.__setattr__(self, attr, value) + object.__setattr__(self, 'int', state['int']) + # is_safe was added in 3.7; it is also omitted when it is "unknown" + object.__setattr__(self, 'is_safe', + SafeUUID(state['is_safe']) + if 'is_safe' in state else SafeUUID.unknown) def __eq__(self, other): if isinstance(other, UUID): From webhook-mailer at python.org Mon Sep 10 11:43:17 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 15:43:17 -0000 Subject: [Python-checkins] bpo-34588: Fix an off-by-one error in traceback formatting. (GH-9077) Message-ID: https://github.com/python/cpython/commit/d545869d084e70d4838310e79b52a25a72a1ca56 commit: d545869d084e70d4838310e79b52a25a72a1ca56 branch: master author: Benjamin Peterson committer: GitHub date: 2018-09-10T08:43:10-07:00 summary: bpo-34588: Fix an off-by-one error in traceback formatting. (GH-9077) The recursive frame pruning code always undercounted the number of elided frames by one. That is, in the "[Previous line repeated N more times]" message, N would always be one too few. Near the recursive pruning cutoff, one frame could be silently dropped. That situation is demonstrated in the OP of the bug report. The fix is to start the identical frame counter at 1. files: A Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst M Lib/test/test_traceback.py M Lib/traceback.py M Python/traceback.c diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index bffc03e663ff..8a3aa8a8648f 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -373,7 +373,7 @@ def g(count=10): ' return g(count-1)\n' f' File "{__file__}", line {lineno_g+2}, in g\n' ' return g(count-1)\n' - ' [Previous line repeated 6 more times]\n' + ' [Previous line repeated 7 more times]\n' f' File "{__file__}", line {lineno_g+3}, in g\n' ' raise ValueError\n' 'ValueError\n' @@ -412,7 +412,7 @@ def h(count=10): ' return h(count-1)\n' f' File "{__file__}", line {lineno_h+2}, in h\n' ' return h(count-1)\n' - ' [Previous line repeated 6 more times]\n' + ' [Previous line repeated 7 more times]\n' f' File "{__file__}", line {lineno_h+3}, in h\n' ' g()\n' ) @@ -420,6 +420,63 @@ def h(count=10): actual = stderr_h.getvalue().splitlines() self.assertEqual(actual, expected) + # Check the boundary conditions. First, test just below the cutoff. + with captured_output("stderr") as stderr_g: + try: + g(traceback._RECURSIVE_CUTOFF) + except ValueError as exc: + render_exc() + else: + self.fail("no error raised") + result_g = ( + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+3}, in g\n' + ' raise ValueError\n' + 'ValueError\n' + ) + tb_line = ( + 'Traceback (most recent call last):\n' + f' File "{__file__}", line {lineno_g+71}, in _check_recursive_traceback_display\n' + ' g(traceback._RECURSIVE_CUTOFF)\n' + ) + expected = (tb_line + result_g).splitlines() + actual = stderr_g.getvalue().splitlines() + self.assertEqual(actual, expected) + + # Second, test just above the cutoff. + with captured_output("stderr") as stderr_g: + try: + g(traceback._RECURSIVE_CUTOFF + 1) + except ValueError as exc: + render_exc() + else: + self.fail("no error raised") + result_g = ( + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + ' [Previous line repeated 1 more time]\n' + f' File "{__file__}", line {lineno_g+3}, in g\n' + ' raise ValueError\n' + 'ValueError\n' + ) + tb_line = ( + 'Traceback (most recent call last):\n' + f' File "{__file__}", line {lineno_g+99}, in _check_recursive_traceback_display\n' + ' g(traceback._RECURSIVE_CUTOFF + 1)\n' + ) + expected = (tb_line + result_g).splitlines() + actual = stderr_g.getvalue().splitlines() + self.assertEqual(actual, expected) + def test_recursive_traceback_python(self): self._check_recursive_traceback_display(traceback.print_exc) diff --git a/Lib/traceback.py b/Lib/traceback.py index afab0a4b91f1..4e7605d15fa4 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -310,6 +310,8 @@ def walk_tb(tb): tb = tb.tb_next +_RECURSIVE_CUTOFF = 3 # Also hardcoded in traceback.c. + class StackSummary(list): """A stack of frames.""" @@ -398,18 +400,21 @@ def format(self): last_name = None count = 0 for frame in self: - if (last_file is not None and last_file == frame.filename and - last_line is not None and last_line == frame.lineno and - last_name is not None and last_name == frame.name): - count += 1 - else: - if count > 3: - result.append(f' [Previous line repeated {count-3} more times]\n') + if (last_file is None or last_file != frame.filename or + last_line is None or last_line != frame.lineno or + last_name is None or last_name != frame.name): + if count > _RECURSIVE_CUTOFF: + count -= _RECURSIVE_CUTOFF + result.append( + f' [Previous line repeated {count} more ' + f'time{"s" if count > 1 else ""}]\n' + ) last_file = frame.filename last_line = frame.lineno last_name = frame.name count = 0 - if count >= 3: + count += 1 + if count > _RECURSIVE_CUTOFF: continue row = [] row.append(' File "{}", line {}, in {}\n'.format( @@ -420,8 +425,12 @@ def format(self): for name, value in sorted(frame.locals.items()): row.append(' {name} = {value}\n'.format(name=name, value=value)) result.append(''.join(row)) - if count > 3: - result.append(f' [Previous line repeated {count-3} more times]\n') + if count > _RECURSIVE_CUTOFF: + count -= _RECURSIVE_CUTOFF + result.append( + f' [Previous line repeated {count} more ' + f'time{"s" if count > 1 else ""}]\n' + ) return result diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst b/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst new file mode 100644 index 000000000000..ec7a57f23ad2 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst @@ -0,0 +1,2 @@ +Fix an off-by-one in the recursive call pruning feature of traceback +formatting. diff --git a/Python/traceback.c b/Python/traceback.c index 21fb03416002..5b5c715a0f44 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -511,16 +511,21 @@ tb_displayline(PyObject *f, PyObject *filename, int lineno, PyObject *name) return err; } +static const int TB_RECURSIVE_CUTOFF = 3; // Also hardcoded in traceback.py. + static int tb_print_line_repeated(PyObject *f, long cnt) { - int err; + cnt -= TB_RECURSIVE_CUTOFF; PyObject *line = PyUnicode_FromFormat( - " [Previous line repeated %ld more times]\n", cnt-3); + (cnt > 1) + ? " [Previous line repeated %ld more times]\n" + : " [Previous line repeated %ld more time]\n", + cnt); if (line == NULL) { return -1; } - err = PyFile_WriteObject(line, f, Py_PRINT_RAW); + int err = PyFile_WriteObject(line, f, Py_PRINT_RAW); Py_DECREF(line); return err; } @@ -544,15 +549,11 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) tb = tb->tb_next; } while (tb != NULL && err == 0) { - if (last_file != NULL && - tb->tb_frame->f_code->co_filename == last_file && - last_line != -1 && tb->tb_lineno == last_line && - last_name != NULL && tb->tb_frame->f_code->co_name == last_name) - { - cnt++; - } - else { - if (cnt > 3) { + if (last_file == NULL || + tb->tb_frame->f_code->co_filename != last_file || + last_line == -1 || tb->tb_lineno != last_line || + last_name == NULL || tb->tb_frame->f_code->co_name != last_name) { + if (cnt > TB_RECURSIVE_CUTOFF) { err = tb_print_line_repeated(f, cnt); } last_file = tb->tb_frame->f_code->co_filename; @@ -560,7 +561,8 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) last_name = tb->tb_frame->f_code->co_name; cnt = 0; } - if (err == 0 && cnt < 3) { + cnt++; + if (err == 0 && cnt <= TB_RECURSIVE_CUTOFF) { err = tb_displayline(f, tb->tb_frame->f_code->co_filename, tb->tb_lineno, @@ -571,7 +573,7 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) } tb = tb->tb_next; } - if (err == 0 && cnt > 3) { + if (err == 0 && cnt > TB_RECURSIVE_CUTOFF) { err = tb_print_line_repeated(f, cnt); } return err; From webhook-mailer at python.org Mon Sep 10 11:47:36 2018 From: webhook-mailer at python.org (Tal Einat) Date: Mon, 10 Sep 2018 15:47:36 -0000 Subject: [Python-checkins] [3.7] bpo-34621: fix uuid.UUID (un)pickling compatbility with older Python versions (<3.7) (GH-9133) Message-ID: https://github.com/python/cpython/commit/d53f1cabe8837697df4acb70c9c6537461b5eeda commit: d53f1cabe8837697df4acb70c9c6537461b5eeda branch: 3.7 author: Tal Einat committer: GitHub date: 2018-09-10T18:47:29+03:00 summary: [3.7] bpo-34621: fix uuid.UUID (un)pickling compatbility with older Python versions (<3.7) (GH-9133) files: A Misc/NEWS.d/next/Library/2018-09-10-15-54-58.bpo-34621.Uqj5x3.rst M Lib/test/test_uuid.py M Lib/uuid.py diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py index 7af1d7aec797..dc502b97ca4e 100644 --- a/Lib/test/test_uuid.py +++ b/Lib/test/test_uuid.py @@ -2,10 +2,13 @@ from test import support import builtins import contextlib +import copy import io import os +import pickle import shutil import subprocess +import sys py_uuid = support.import_fresh_module('uuid', blocked=['_uuid']) c_uuid = support.import_fresh_module('uuid', fresh=['_uuid']) @@ -311,6 +314,140 @@ def test_getnode(self): node2 = self.uuid.getnode() self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2)) + def test_pickle_roundtrip(self): + def check(actual, expected): + self.assertEqual(actual, expected) + self.assertEqual(actual.is_safe, expected.is_safe) + + with support.swap_item(sys.modules, 'uuid', self.uuid): + for is_safe in self.uuid.SafeUUID: + u = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5', + is_safe=is_safe) + check(copy.copy(u), u) + check(copy.deepcopy(u), u) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(protocol=proto): + check(pickle.loads(pickle.dumps(u, proto)), u) + + def test_unpickle_previous_python_versions(self): + def check(actual, expected): + self.assertEqual(actual, expected) + self.assertEqual(actual.is_safe, expected.is_safe) + + pickled_uuids = [ + # Python 2.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dS\'int\'\nL287307832597519156748809049798316161701L\nsb.', + # Python 2.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}U\x03intL287307832597519156748809049798316161701L\nsb.', + # Python 2.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}U\x03int\x8a\x11\xa5z\xecz\nI\xdf}' + b'\xde\xa0Bf\xcey%\xd8\x00sb.', + # Python 3.6, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsb.', + # Python 3.6, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}X\x03\x00\x00\x00intL287307832597519156748809049798316161701L' + b'\nsb.', + # Python 3.6, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}X\x03\x00\x00\x00int\x8a\x11\xa5z\xec' + b'z\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00sb.', + # Python 3.6, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}X\x03\x00\x00\x00int\x8a\x11\xa5z\xec' + b'z\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00sb.', + # Python 3.6, protocol 4 + b'\x80\x04\x95+\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x8c\x04UUI' + b'D\x93)\x81}\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0Bf\xcey%' + b'\xd8\x00sb.', + # Python 3.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsVis_safe\n' + b'cuuid\nSafeUUID\n(NtRsb.', + # Python 3.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}(X\x03\x00\x00\x00intL287307832597519156748809049798316161701' + b'L\nX\x07\x00\x00\x00is_safecuuid\nSafeUUID\n(NtRub.', + # Python 3.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nN\x85Rub.', + # Python 3.7, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nN\x85Rub.', + # Python 3.7, protocol 4 + b'\x80\x04\x95F\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c' + b'\x04UUID\x93)\x81}(\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0' + b'Bf\xcey%\xd8\x00\x8c\x07is_safeh\x00\x8c\x08SafeUUID\x93N\x85Rub' + b'.', + ] + pickled_uuids_safe = [ + # Python 3.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsVis_safe\n' + b'cuuid\nSafeUUID\n(I0\ntRsb.', + # Python 3.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}(X\x03\x00\x00\x00intL287307832597519156748809049798316161701' + b'L\nX\x07\x00\x00\x00is_safecuuid\nSafeUUID\n(K\x00tRub.', + # Python 3.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nK\x00\x85Rub.', + # Python 3.7, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nK\x00\x85Rub.', + # Python 3.7, protocol 4 + b'\x80\x04\x95G\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c' + b'\x04UUID\x93)\x81}(\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0' + b'Bf\xcey%\xd8\x00\x8c\x07is_safeh\x00\x8c\x08SafeUUID\x93K\x00' + b'\x85Rub.', + ] + pickled_uuids_unsafe = [ + # Python 3.7, protocol 0 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR(dVint\nL287307832597519156748809049798316161701L\nsVis_safe\n' + b'cuuid\nSafeUUID\n(I-1\ntRsb.', + # Python 3.7, protocol 1 + b'ccopy_reg\n_reconstructor\n(cuuid\nUUID\nc__builtin__\nobject\nN' + b'tR}(X\x03\x00\x00\x00intL287307832597519156748809049798316161701' + b'L\nX\x07\x00\x00\x00is_safecuuid\nSafeUUID\n(J\xff\xff\xff\xfftR' + b'ub.', + # Python 3.7, protocol 2 + b'\x80\x02cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nJ\xff\xff\xff\xff\x85Rub.', + # Python 3.7, protocol 3 + b'\x80\x03cuuid\nUUID\n)\x81}(X\x03\x00\x00\x00int\x8a\x11\xa5z' + b'\xecz\nI\xdf}\xde\xa0Bf\xcey%\xd8\x00X\x07\x00\x00\x00is_safecuu' + b'id\nSafeUUID\nJ\xff\xff\xff\xff\x85Rub.', + # Python 3.7, protocol 4 + b'\x80\x04\x95J\x00\x00\x00\x00\x00\x00\x00\x8c\x04uuid\x94\x8c' + b'\x04UUID\x93)\x81}(\x8c\x03int\x8a\x11\xa5z\xecz\nI\xdf}\xde\xa0' + b'Bf\xcey%\xd8\x00\x8c\x07is_safeh\x00\x8c\x08SafeUUID\x93J\xff' + b'\xff\xff\xff\x85Rub.', + ] + + u = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5') + u_safe = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5', + is_safe=self.uuid.SafeUUID.safe) + u_unsafe = self.uuid.UUID('d82579ce6642a0de7ddf490a7aec7aa5', + is_safe=self.uuid.SafeUUID.unsafe) + + with support.swap_item(sys.modules, 'uuid', self.uuid): + for pickled in pickled_uuids: + # is_safe was added in 3.7. When unpickling values from older + # versions, is_safe will be missing, so it should be set to + # SafeUUID.unknown. + check(pickle.loads(pickled), u) + for pickled in pickled_uuids_safe: + check(pickle.loads(pickled), u_safe) + for pickled in pickled_uuids_unsafe: + check(pickle.loads(pickled), u_unsafe) + # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers # need not necessarily be 48 bits (e.g., EUI-64). def test_uuid1_eui64(self): diff --git a/Lib/uuid.py b/Lib/uuid.py index 66383218e70c..26faa1accd09 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -204,6 +204,23 @@ def __init__(self, hex=None, bytes=None, bytes_le=None, fields=None, self.__dict__['int'] = int self.__dict__['is_safe'] = is_safe + def __getstate__(self): + state = self.__dict__ + if self.is_safe != SafeUUID.unknown: + # is_safe is a SafeUUID instance. Return just its value, so that + # it can be un-pickled in older Python versions without SafeUUID. + state = state.copy() + state['is_safe'] = self.is_safe.value + return state + + def __setstate__(self, state): + self.__dict__.update(state) + # is_safe was added in 3.7; it is also omitted when it is "unknown" + self.__dict__['is_safe'] = ( + SafeUUID(state['is_safe']) + if 'is_safe' in state else SafeUUID.unknown + ) + def __eq__(self, other): if isinstance(other, UUID): return self.int == other.int diff --git a/Misc/NEWS.d/next/Library/2018-09-10-15-54-58.bpo-34621.Uqj5x3.rst b/Misc/NEWS.d/next/Library/2018-09-10-15-54-58.bpo-34621.Uqj5x3.rst new file mode 100644 index 000000000000..1fd60608aea3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-15-54-58.bpo-34621.Uqj5x3.rst @@ -0,0 +1,2 @@ +Fix un/pickling compatbility of uuid.UUID objects with older versions of +Python (<3.7). From webhook-mailer at python.org Mon Sep 10 12:00:13 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 16:00:13 -0000 Subject: [Python-checkins] bpo-34588: Fix an off-by-one error in traceback formatting. (GH-9077) Message-ID: https://github.com/python/cpython/commit/49020174305ca3dc90a811b03a05f44873297c61 commit: 49020174305ca3dc90a811b03a05f44873297c61 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T09:00:08-07:00 summary: bpo-34588: Fix an off-by-one error in traceback formatting. (GH-9077) The recursive frame pruning code always undercounted the number of elided frames by one. That is, in the "[Previous line repeated N more times]" message, N would always be one too few. Near the recursive pruning cutoff, one frame could be silently dropped. That situation is demonstrated in the OP of the bug report. The fix is to start the identical frame counter at 1. (cherry picked from commit d545869d084e70d4838310e79b52a25a72a1ca56) Co-authored-by: Benjamin Peterson files: A Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst M Lib/test/test_traceback.py M Lib/traceback.py M Python/traceback.c diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index bffc03e663ff..8a3aa8a8648f 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -373,7 +373,7 @@ def g(count=10): ' return g(count-1)\n' f' File "{__file__}", line {lineno_g+2}, in g\n' ' return g(count-1)\n' - ' [Previous line repeated 6 more times]\n' + ' [Previous line repeated 7 more times]\n' f' File "{__file__}", line {lineno_g+3}, in g\n' ' raise ValueError\n' 'ValueError\n' @@ -412,7 +412,7 @@ def h(count=10): ' return h(count-1)\n' f' File "{__file__}", line {lineno_h+2}, in h\n' ' return h(count-1)\n' - ' [Previous line repeated 6 more times]\n' + ' [Previous line repeated 7 more times]\n' f' File "{__file__}", line {lineno_h+3}, in h\n' ' g()\n' ) @@ -420,6 +420,63 @@ def h(count=10): actual = stderr_h.getvalue().splitlines() self.assertEqual(actual, expected) + # Check the boundary conditions. First, test just below the cutoff. + with captured_output("stderr") as stderr_g: + try: + g(traceback._RECURSIVE_CUTOFF) + except ValueError as exc: + render_exc() + else: + self.fail("no error raised") + result_g = ( + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+3}, in g\n' + ' raise ValueError\n' + 'ValueError\n' + ) + tb_line = ( + 'Traceback (most recent call last):\n' + f' File "{__file__}", line {lineno_g+71}, in _check_recursive_traceback_display\n' + ' g(traceback._RECURSIVE_CUTOFF)\n' + ) + expected = (tb_line + result_g).splitlines() + actual = stderr_g.getvalue().splitlines() + self.assertEqual(actual, expected) + + # Second, test just above the cutoff. + with captured_output("stderr") as stderr_g: + try: + g(traceback._RECURSIVE_CUTOFF + 1) + except ValueError as exc: + render_exc() + else: + self.fail("no error raised") + result_g = ( + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + ' [Previous line repeated 1 more time]\n' + f' File "{__file__}", line {lineno_g+3}, in g\n' + ' raise ValueError\n' + 'ValueError\n' + ) + tb_line = ( + 'Traceback (most recent call last):\n' + f' File "{__file__}", line {lineno_g+99}, in _check_recursive_traceback_display\n' + ' g(traceback._RECURSIVE_CUTOFF + 1)\n' + ) + expected = (tb_line + result_g).splitlines() + actual = stderr_g.getvalue().splitlines() + self.assertEqual(actual, expected) + def test_recursive_traceback_python(self): self._check_recursive_traceback_display(traceback.print_exc) diff --git a/Lib/traceback.py b/Lib/traceback.py index afab0a4b91f1..4e7605d15fa4 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -310,6 +310,8 @@ def walk_tb(tb): tb = tb.tb_next +_RECURSIVE_CUTOFF = 3 # Also hardcoded in traceback.c. + class StackSummary(list): """A stack of frames.""" @@ -398,18 +400,21 @@ def format(self): last_name = None count = 0 for frame in self: - if (last_file is not None and last_file == frame.filename and - last_line is not None and last_line == frame.lineno and - last_name is not None and last_name == frame.name): - count += 1 - else: - if count > 3: - result.append(f' [Previous line repeated {count-3} more times]\n') + if (last_file is None or last_file != frame.filename or + last_line is None or last_line != frame.lineno or + last_name is None or last_name != frame.name): + if count > _RECURSIVE_CUTOFF: + count -= _RECURSIVE_CUTOFF + result.append( + f' [Previous line repeated {count} more ' + f'time{"s" if count > 1 else ""}]\n' + ) last_file = frame.filename last_line = frame.lineno last_name = frame.name count = 0 - if count >= 3: + count += 1 + if count > _RECURSIVE_CUTOFF: continue row = [] row.append(' File "{}", line {}, in {}\n'.format( @@ -420,8 +425,12 @@ def format(self): for name, value in sorted(frame.locals.items()): row.append(' {name} = {value}\n'.format(name=name, value=value)) result.append(''.join(row)) - if count > 3: - result.append(f' [Previous line repeated {count-3} more times]\n') + if count > _RECURSIVE_CUTOFF: + count -= _RECURSIVE_CUTOFF + result.append( + f' [Previous line repeated {count} more ' + f'time{"s" if count > 1 else ""}]\n' + ) return result diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst b/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst new file mode 100644 index 000000000000..ec7a57f23ad2 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst @@ -0,0 +1,2 @@ +Fix an off-by-one in the recursive call pruning feature of traceback +formatting. diff --git a/Python/traceback.c b/Python/traceback.c index b00864b06e43..95bef64e734b 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -511,16 +511,21 @@ tb_displayline(PyObject *f, PyObject *filename, int lineno, PyObject *name) return err; } +static const int TB_RECURSIVE_CUTOFF = 3; // Also hardcoded in traceback.py. + static int tb_print_line_repeated(PyObject *f, long cnt) { - int err; + cnt -= TB_RECURSIVE_CUTOFF; PyObject *line = PyUnicode_FromFormat( - " [Previous line repeated %ld more times]\n", cnt-3); + (cnt > 1) + ? " [Previous line repeated %ld more times]\n" + : " [Previous line repeated %ld more time]\n", + cnt); if (line == NULL) { return -1; } - err = PyFile_WriteObject(line, f, Py_PRINT_RAW); + int err = PyFile_WriteObject(line, f, Py_PRINT_RAW); Py_DECREF(line); return err; } @@ -544,15 +549,11 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) tb = tb->tb_next; } while (tb != NULL && err == 0) { - if (last_file != NULL && - tb->tb_frame->f_code->co_filename == last_file && - last_line != -1 && tb->tb_lineno == last_line && - last_name != NULL && tb->tb_frame->f_code->co_name == last_name) - { - cnt++; - } - else { - if (cnt > 3) { + if (last_file == NULL || + tb->tb_frame->f_code->co_filename != last_file || + last_line == -1 || tb->tb_lineno != last_line || + last_name == NULL || tb->tb_frame->f_code->co_name != last_name) { + if (cnt > TB_RECURSIVE_CUTOFF) { err = tb_print_line_repeated(f, cnt); } last_file = tb->tb_frame->f_code->co_filename; @@ -560,7 +561,8 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) last_name = tb->tb_frame->f_code->co_name; cnt = 0; } - if (err == 0 && cnt < 3) { + cnt++; + if (err == 0 && cnt <= TB_RECURSIVE_CUTOFF) { err = tb_displayline(f, tb->tb_frame->f_code->co_filename, tb->tb_lineno, @@ -571,7 +573,7 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) } tb = tb->tb_next; } - if (err == 0 && cnt > 3) { + if (err == 0 && cnt > TB_RECURSIVE_CUTOFF) { err = tb_print_line_repeated(f, cnt); } return err; From webhook-mailer at python.org Mon Sep 10 12:10:25 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 16:10:25 -0000 Subject: [Python-checkins] bpo-34588: Fix an off-by-one error in traceback formatting. (GH-9077) Message-ID: https://github.com/python/cpython/commit/afb25bc2b5767ac3a83bc8c4d2826e8fdcb6b0e7 commit: afb25bc2b5767ac3a83bc8c4d2826e8fdcb6b0e7 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T09:10:21-07:00 summary: bpo-34588: Fix an off-by-one error in traceback formatting. (GH-9077) The recursive frame pruning code always undercounted the number of elided frames by one. That is, in the "[Previous line repeated N more times]" message, N would always be one too few. Near the recursive pruning cutoff, one frame could be silently dropped. That situation is demonstrated in the OP of the bug report. The fix is to start the identical frame counter at 1. (cherry picked from commit d545869d084e70d4838310e79b52a25a72a1ca56) Co-authored-by: Benjamin Peterson files: A Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst M Lib/test/test_traceback.py M Lib/traceback.py M Python/traceback.c diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index bffc03e663ff..8a3aa8a8648f 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -373,7 +373,7 @@ def g(count=10): ' return g(count-1)\n' f' File "{__file__}", line {lineno_g+2}, in g\n' ' return g(count-1)\n' - ' [Previous line repeated 6 more times]\n' + ' [Previous line repeated 7 more times]\n' f' File "{__file__}", line {lineno_g+3}, in g\n' ' raise ValueError\n' 'ValueError\n' @@ -412,7 +412,7 @@ def h(count=10): ' return h(count-1)\n' f' File "{__file__}", line {lineno_h+2}, in h\n' ' return h(count-1)\n' - ' [Previous line repeated 6 more times]\n' + ' [Previous line repeated 7 more times]\n' f' File "{__file__}", line {lineno_h+3}, in h\n' ' g()\n' ) @@ -420,6 +420,63 @@ def h(count=10): actual = stderr_h.getvalue().splitlines() self.assertEqual(actual, expected) + # Check the boundary conditions. First, test just below the cutoff. + with captured_output("stderr") as stderr_g: + try: + g(traceback._RECURSIVE_CUTOFF) + except ValueError as exc: + render_exc() + else: + self.fail("no error raised") + result_g = ( + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+3}, in g\n' + ' raise ValueError\n' + 'ValueError\n' + ) + tb_line = ( + 'Traceback (most recent call last):\n' + f' File "{__file__}", line {lineno_g+71}, in _check_recursive_traceback_display\n' + ' g(traceback._RECURSIVE_CUTOFF)\n' + ) + expected = (tb_line + result_g).splitlines() + actual = stderr_g.getvalue().splitlines() + self.assertEqual(actual, expected) + + # Second, test just above the cutoff. + with captured_output("stderr") as stderr_g: + try: + g(traceback._RECURSIVE_CUTOFF + 1) + except ValueError as exc: + render_exc() + else: + self.fail("no error raised") + result_g = ( + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + f' File "{__file__}", line {lineno_g+2}, in g\n' + ' return g(count-1)\n' + ' [Previous line repeated 1 more time]\n' + f' File "{__file__}", line {lineno_g+3}, in g\n' + ' raise ValueError\n' + 'ValueError\n' + ) + tb_line = ( + 'Traceback (most recent call last):\n' + f' File "{__file__}", line {lineno_g+99}, in _check_recursive_traceback_display\n' + ' g(traceback._RECURSIVE_CUTOFF + 1)\n' + ) + expected = (tb_line + result_g).splitlines() + actual = stderr_g.getvalue().splitlines() + self.assertEqual(actual, expected) + def test_recursive_traceback_python(self): self._check_recursive_traceback_display(traceback.print_exc) diff --git a/Lib/traceback.py b/Lib/traceback.py index d2b102b73c40..1e5190954b95 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -311,6 +311,8 @@ def walk_tb(tb): tb = tb.tb_next +_RECURSIVE_CUTOFF = 3 # Also hardcoded in traceback.c. + class StackSummary(list): """A stack of frames.""" @@ -399,18 +401,21 @@ def format(self): last_name = None count = 0 for frame in self: - if (last_file is not None and last_file == frame.filename and - last_line is not None and last_line == frame.lineno and - last_name is not None and last_name == frame.name): - count += 1 - else: - if count > 3: - result.append(f' [Previous line repeated {count-3} more times]\n') + if (last_file is None or last_file != frame.filename or + last_line is None or last_line != frame.lineno or + last_name is None or last_name != frame.name): + if count > _RECURSIVE_CUTOFF: + count -= _RECURSIVE_CUTOFF + result.append( + f' [Previous line repeated {count} more ' + f'time{"s" if count > 1 else ""}]\n' + ) last_file = frame.filename last_line = frame.lineno last_name = frame.name count = 0 - if count >= 3: + count += 1 + if count > _RECURSIVE_CUTOFF: continue row = [] row.append(' File "{}", line {}, in {}\n'.format( @@ -421,8 +426,12 @@ def format(self): for name, value in sorted(frame.locals.items()): row.append(' {name} = {value}\n'.format(name=name, value=value)) result.append(''.join(row)) - if count > 3: - result.append(f' [Previous line repeated {count-3} more times]\n') + if count > _RECURSIVE_CUTOFF: + count -= _RECURSIVE_CUTOFF + result.append( + f' [Previous line repeated {count} more ' + f'time{"s" if count > 1 else ""}]\n' + ) return result diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst b/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst new file mode 100644 index 000000000000..ec7a57f23ad2 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-09-05-22-56-52.bpo-34588.UIuPmL.rst @@ -0,0 +1,2 @@ +Fix an off-by-one in the recursive call pruning feature of traceback +formatting. diff --git a/Python/traceback.c b/Python/traceback.c index d9620675f747..145d028ba353 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -413,16 +413,21 @@ tb_displayline(PyObject *f, PyObject *filename, int lineno, PyObject *name) return err; } +static const int TB_RECURSIVE_CUTOFF = 3; // Also hardcoded in traceback.py. + static int tb_print_line_repeated(PyObject *f, long cnt) { - int err; + cnt -= TB_RECURSIVE_CUTOFF; PyObject *line = PyUnicode_FromFormat( - " [Previous line repeated %ld more times]\n", cnt-3); + (cnt > 1) + ? " [Previous line repeated %ld more times]\n" + : " [Previous line repeated %ld more time]\n", + cnt); if (line == NULL) { return -1; } - err = PyFile_WriteObject(line, f, Py_PRINT_RAW); + int err = PyFile_WriteObject(line, f, Py_PRINT_RAW); Py_DECREF(line); return err; } @@ -446,15 +451,11 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) tb = tb->tb_next; } while (tb != NULL && err == 0) { - if (last_file != NULL && - tb->tb_frame->f_code->co_filename == last_file && - last_line != -1 && tb->tb_lineno == last_line && - last_name != NULL && tb->tb_frame->f_code->co_name == last_name) - { - cnt++; - } - else { - if (cnt > 3) { + if (last_file == NULL || + tb->tb_frame->f_code->co_filename != last_file || + last_line == -1 || tb->tb_lineno != last_line || + last_name == NULL || tb->tb_frame->f_code->co_name != last_name) { + if (cnt > TB_RECURSIVE_CUTOFF) { err = tb_print_line_repeated(f, cnt); } last_file = tb->tb_frame->f_code->co_filename; @@ -462,7 +463,8 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) last_name = tb->tb_frame->f_code->co_name; cnt = 0; } - if (err == 0 && cnt < 3) { + cnt++; + if (err == 0 && cnt <= TB_RECURSIVE_CUTOFF) { err = tb_displayline(f, tb->tb_frame->f_code->co_filename, tb->tb_lineno, @@ -473,7 +475,7 @@ tb_printinternal(PyTracebackObject *tb, PyObject *f, long limit) } tb = tb->tb_next; } - if (err == 0 && cnt > 3) { + if (err == 0 && cnt > TB_RECURSIVE_CUTOFF) { err = tb_print_line_repeated(f, cnt); } return err; From webhook-mailer at python.org Mon Sep 10 12:39:53 2018 From: webhook-mailer at python.org (Ethan Furman) Date: Mon, 10 Sep 2018 16:39:53 -0000 Subject: [Python-checkins] [3.7] bpo-34282: Fix Enum._convert method shadowing members named _convert (GH-9034) Message-ID: https://github.com/python/cpython/commit/c0d63bf73b35df374e6e66c08b0e297fb828d744 commit: c0d63bf73b35df374e6e66c08b0e297fb828d744 branch: 3.7 author: orlnub123 committer: Ethan Furman date: 2018-09-10T09:39:48-07:00 summary: [3.7] bpo-34282: Fix Enum._convert method shadowing members named _convert (GH-9034) * Fix Enum._convert shadowing members named _convert files: A Misc/NEWS.d/next/Library/2018-09-02-13-33-35.bpo-34282.ztyXH8.rst M Lib/enum.py M Lib/test/test_enum.py diff --git a/Lib/enum.py b/Lib/enum.py index 576de031805d..69b41fe7cbc0 100644 --- a/Lib/enum.py +++ b/Lib/enum.py @@ -171,9 +171,11 @@ def __new__(metacls, cls, bases, classdict): enum_class._member_map_ = OrderedDict() # name->value map enum_class._member_type_ = member_type - # save attributes from super classes so we know if we can take - # the shortcut of storing members in the class dict - base_attributes = {a for b in enum_class.mro() for a in b.__dict__} + # save DynamicClassAttribute attributes from super classes so we know + # if we can take the shortcut of storing members in the class dict + dynamic_attributes = {k for c in enum_class.mro() + for k, v in c.__dict__.items() + if isinstance(v, DynamicClassAttribute)} # Reverse value->name map for hashable values. enum_class._value2member_map_ = {} @@ -233,7 +235,7 @@ def __new__(metacls, cls, bases, classdict): enum_class._member_names_.append(member_name) # performance boost for any member that would not shadow # a DynamicClassAttribute - if member_name not in base_attributes: + if member_name not in dynamic_attributes: setattr(enum_class, member_name, enum_member) # now add to _member_map_ enum_class._member_map_[member_name] = enum_member diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py index ef2d1daaf942..4b1722894601 100644 --- a/Lib/test/test_enum.py +++ b/Lib/test/test_enum.py @@ -1516,6 +1516,23 @@ class MoreColor(Color): yellow = 6 self.assertEqual(MoreColor.magenta.hex(), '5 hexlified!') + def test_subclass_duplicate_name(self): + class Base(Enum): + def test(self): + pass + class Test(Base): + test = 1 + self.assertIs(type(Test.test), Test) + + def test_subclass_duplicate_name_dynamic(self): + from types import DynamicClassAttribute + class Base(Enum): + @DynamicClassAttribute + def test(self): + return 'dynamic' + class Test(Base): + test = 1 + self.assertEqual(Test.test.test, 'dynamic') def test_no_duplicates(self): class UniqueEnum(Enum): diff --git a/Misc/NEWS.d/next/Library/2018-09-02-13-33-35.bpo-34282.ztyXH8.rst b/Misc/NEWS.d/next/Library/2018-09-02-13-33-35.bpo-34282.ztyXH8.rst new file mode 100644 index 000000000000..c1e606ab0ce5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-02-13-33-35.bpo-34282.ztyXH8.rst @@ -0,0 +1 @@ +Fix enum members getting shadowed by parent attributes. From webhook-mailer at python.org Mon Sep 10 12:41:16 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 16:41:16 -0000 Subject: [Python-checkins] bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) Message-ID: https://github.com/python/cpython/commit/f51a46631f8dcca596c08a934a766da9afe93c06 commit: f51a46631f8dcca596c08a934a766da9afe93c06 branch: 2.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T09:41:12-07:00 summary: bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) (cherry picked from commit 4e519377b1b84c9414a360961276993d24198825) Co-authored-by: Zackery Spytz files: M PC/_msi.c diff --git a/PC/_msi.c b/PC/_msi.c index d7700f09c749..68c4e79e2945 100644 --- a/PC/_msi.c +++ b/PC/_msi.c @@ -321,6 +321,10 @@ msierror(int status) code = MsiRecordGetInteger(err, 1); /* XXX code */ if (MsiFormatRecord(0, err, res, &size) == ERROR_MORE_DATA) { res = malloc(size+1); + if (res == NULL) { + MsiCloseHandle(err); + return PyErr_NoMemory(); + } MsiFormatRecord(0, err, res, &size); res[size]='\0'; } @@ -544,6 +548,9 @@ summary_getproperty(msiobj* si, PyObject *args) &fval, sval, &ssize); if (status == ERROR_MORE_DATA) { sval = malloc(ssize); + if (sval == NULL) { + return PyErr_NoMemory(); + } status = MsiSummaryInfoGetProperty(si->h, field, &type, &ival, &fval, sval, &ssize); } From webhook-mailer at python.org Mon Sep 10 12:41:36 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 16:41:36 -0000 Subject: [Python-checkins] bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) Message-ID: https://github.com/python/cpython/commit/8a0c254fdd68cfafede168356fc5c5c3e372bc3f commit: 8a0c254fdd68cfafede168356fc5c5c3e372bc3f branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T09:41:31-07:00 summary: bpo-23855: Add missing NULL checks for malloc() in _msi.c (GH-9038) (cherry picked from commit 4e519377b1b84c9414a360961276993d24198825) Co-authored-by: Zackery Spytz files: M PC/_msi.c diff --git a/PC/_msi.c b/PC/_msi.c index a7c8fa3d092a..0487743b59aa 100644 --- a/PC/_msi.c +++ b/PC/_msi.c @@ -340,6 +340,10 @@ msierror(int status) code = MsiRecordGetInteger(err, 1); /* XXX code */ if (MsiFormatRecord(0, err, res, &size) == ERROR_MORE_DATA) { res = malloc(size+1); + if (res == NULL) { + MsiCloseHandle(err); + return PyErr_NoMemory(); + } MsiFormatRecord(0, err, res, &size); res[size]='\0'; } @@ -563,6 +567,9 @@ summary_getproperty(msiobj* si, PyObject *args) &fval, sval, &ssize); if (status == ERROR_MORE_DATA) { sval = malloc(ssize); + if (sval == NULL) { + return PyErr_NoMemory(); + } status = MsiSummaryInfoGetProperty(si->h, field, &type, &ival, &fval, sval, &ssize); } From webhook-mailer at python.org Mon Sep 10 12:46:12 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 16:46:12 -0000 Subject: [Python-checkins] Fix misleading mentions of tp_size in comments (GH-9093) Message-ID: https://github.com/python/cpython/commit/0e0bc4e221f592f305d335faf5f8046484eb9238 commit: 0e0bc4e221f592f305d335faf5f8046484eb9238 branch: master author: Peter Eisentraut committer: Benjamin Peterson date: 2018-09-10T09:46:08-07:00 summary: Fix misleading mentions of tp_size in comments (GH-9093) Many type object initializations labeled a field "tp_size" in the comment, but the name of that field is tp_basicsize. files: M Modules/_abc.c M Modules/_blake2/blake2b_impl.c M Modules/_blake2/blake2s_impl.c M Modules/_queuemodule.c M Modules/_sha3/sha3module.c M Modules/_threadmodule.c M Modules/_xxsubinterpretersmodule.c M Modules/md5module.c M Modules/mmapmodule.c M Modules/ossaudiodev.c M Modules/sha1module.c M Modules/sha256module.c M Modules/sha512module.c M Objects/moduleobject.c M Objects/namespaceobject.c diff --git a/Modules/_abc.c b/Modules/_abc.c index ce9140fd03cc..9de199fa143f 100644 --- a/Modules/_abc.c +++ b/Modules/_abc.c @@ -68,7 +68,7 @@ PyDoc_STRVAR(abc_data_doc, static PyTypeObject _abc_data_type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_abc_data", /*tp_name*/ - sizeof(_abc_data), /*tp_size*/ + sizeof(_abc_data), /*tp_basicsize*/ .tp_dealloc = (destructor)abc_data_dealloc, .tp_flags = Py_TPFLAGS_DEFAULT, .tp_alloc = PyType_GenericAlloc, diff --git a/Modules/_blake2/blake2b_impl.c b/Modules/_blake2/blake2b_impl.c index 7ad4b650b325..788c15c31d8c 100644 --- a/Modules/_blake2/blake2b_impl.c +++ b/Modules/_blake2/blake2b_impl.c @@ -398,7 +398,7 @@ py_blake2b_dealloc(PyObject *self) PyTypeObject PyBlake2_BLAKE2bType = { PyVarObject_HEAD_INIT(NULL, 0) "_blake2.blake2b", /* tp_name */ - sizeof(BLAKE2bObject), /* tp_size */ + sizeof(BLAKE2bObject), /* tp_basicsize */ 0, /* tp_itemsize */ py_blake2b_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/_blake2/blake2s_impl.c b/Modules/_blake2/blake2s_impl.c index 8864fd829787..c8bcedeabd57 100644 --- a/Modules/_blake2/blake2s_impl.c +++ b/Modules/_blake2/blake2s_impl.c @@ -398,7 +398,7 @@ py_blake2s_dealloc(PyObject *self) PyTypeObject PyBlake2_BLAKE2sType = { PyVarObject_HEAD_INIT(NULL, 0) "_blake2.blake2s", /* tp_name */ - sizeof(BLAKE2sObject), /* tp_size */ + sizeof(BLAKE2sObject), /* tp_basicsize */ 0, /* tp_itemsize */ py_blake2s_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/_queuemodule.c b/Modules/_queuemodule.c index 0eb993027459..2a9406cc4760 100644 --- a/Modules/_queuemodule.c +++ b/Modules/_queuemodule.c @@ -309,7 +309,7 @@ static PyMethodDef simplequeue_methods[] = { static PyTypeObject PySimpleQueueType = { PyVarObject_HEAD_INIT(NULL, 0) "_queue.SimpleQueue", /*tp_name*/ - sizeof(simplequeueobject), /*tp_size*/ + sizeof(simplequeueobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)simplequeue_dealloc, /*tp_dealloc*/ diff --git a/Modules/_sha3/sha3module.c b/Modules/_sha3/sha3module.c index f5032fcb72dd..46c1ff153852 100644 --- a/Modules/_sha3/sha3module.c +++ b/Modules/_sha3/sha3module.c @@ -489,7 +489,7 @@ static PyGetSetDef SHA3_getseters[] = { static PyTypeObject type_obj = { \ PyVarObject_HEAD_INIT(NULL, 0) \ type_name, /* tp_name */ \ - sizeof(SHA3object), /* tp_size */ \ + sizeof(SHA3object), /* tp_basicsize */ \ 0, /* tp_itemsize */ \ /* methods */ \ (destructor)SHA3_dealloc, /* tp_dealloc */ \ diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index f6b39defbc80..d33fa9924308 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -226,7 +226,7 @@ static PyMethodDef lock_methods[] = { static PyTypeObject Locktype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_thread.lock", /*tp_name*/ - sizeof(lockobject), /*tp_size*/ + sizeof(lockobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)lock_dealloc, /*tp_dealloc*/ @@ -487,7 +487,7 @@ static PyMethodDef rlock_methods[] = { static PyTypeObject RLocktype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_thread.RLock", /*tp_name*/ - sizeof(rlockobject), /*tp_size*/ + sizeof(rlockobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)rlock_dealloc, /*tp_dealloc*/ diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c index 7b2cda218309..2eb87871694d 100644 --- a/Modules/_xxsubinterpretersmodule.c +++ b/Modules/_xxsubinterpretersmodule.c @@ -1764,7 +1764,7 @@ PyDoc_STRVAR(channelid_doc, static PyTypeObject ChannelIDtype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_xxsubinterpreters.ChannelID", /* tp_name */ - sizeof(channelid), /* tp_size */ + sizeof(channelid), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)channelid_dealloc, /* tp_dealloc */ 0, /* tp_print */ @@ -2173,7 +2173,7 @@ PyDoc_STRVAR(interpid_doc, static PyTypeObject InterpreterIDtype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "interpreters.InterpreterID", /* tp_name */ - sizeof(interpid), /* tp_size */ + sizeof(interpid), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)interpid_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/md5module.c b/Modules/md5module.c index b019f8287684..c66b27308f05 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -465,7 +465,7 @@ static PyGetSetDef MD5_getseters[] = { static PyTypeObject MD5type = { PyVarObject_HEAD_INIT(NULL, 0) "_md5.md5", /*tp_name*/ - sizeof(MD5object), /*tp_size*/ + sizeof(MD5object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ MD5_dealloc, /*tp_dealloc*/ diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index af9cd7e2be8c..86632358b7fc 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -983,7 +983,7 @@ To map anonymous memory, pass -1 as the fileno (both versions)."); static PyTypeObject mmap_object_type = { PyVarObject_HEAD_INIT(NULL, 0) "mmap.mmap", /* tp_name */ - sizeof(mmap_object), /* tp_size */ + sizeof(mmap_object), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor) mmap_object_dealloc, /* tp_dealloc */ diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 58ee71f90108..150a14eb388a 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -964,7 +964,7 @@ oss_getattro(oss_audio_t *self, PyObject *nameobj) static PyTypeObject OSSAudioType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_audio_device", /*tp_name*/ - sizeof(oss_audio_t), /*tp_size*/ + sizeof(oss_audio_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_dealloc, /*tp_dealloc*/ @@ -996,7 +996,7 @@ static PyTypeObject OSSAudioType = { static PyTypeObject OSSMixerType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_mixer_device", /*tp_name*/ - sizeof(oss_mixer_t), /*tp_size*/ + sizeof(oss_mixer_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_mixer_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha1module.c b/Modules/sha1module.c index d39190b4d5e6..358cd1e7f9ac 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -442,7 +442,7 @@ static PyGetSetDef SHA1_getseters[] = { static PyTypeObject SHA1type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha1.sha1", /*tp_name*/ - sizeof(SHA1object), /*tp_size*/ + sizeof(SHA1object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA1_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha256module.c b/Modules/sha256module.c index e4cb3286ce95..3599eaf8be92 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -529,7 +529,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA224type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha224", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ @@ -563,7 +563,7 @@ static PyTypeObject SHA224type = { static PyTypeObject SHA256type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha256", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha512module.c b/Modules/sha512module.c index 5ac2a2a61cf5..d8c846a450f2 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -594,7 +594,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA384type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha384", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ @@ -628,7 +628,7 @@ static PyTypeObject SHA384type = { static PyTypeObject SHA512type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha512", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index 2156ca0765a0..ccf5f8e6d15a 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -35,7 +35,7 @@ bad_traverse_test(PyObject *self, void *arg) { PyTypeObject PyModuleDef_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "moduledef", /* tp_name */ - sizeof(struct PyModuleDef), /* tp_size */ + sizeof(struct PyModuleDef), /* tp_basicsize */ 0, /* tp_itemsize */ }; @@ -790,7 +790,7 @@ static PyMethodDef module_methods[] = { PyTypeObject PyModule_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "module", /* tp_name */ - sizeof(PyModuleObject), /* tp_size */ + sizeof(PyModuleObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)module_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Objects/namespaceobject.c b/Objects/namespaceobject.c index a6c941afc0ad..0b902693849d 100644 --- a/Objects/namespaceobject.c +++ b/Objects/namespaceobject.c @@ -201,7 +201,7 @@ SimpleNamespace(**kwargs)"); PyTypeObject _PyNamespace_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "types.SimpleNamespace", /* tp_name */ - sizeof(_PyNamespaceObject), /* tp_size */ + sizeof(_PyNamespaceObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)namespace_dealloc, /* tp_dealloc */ 0, /* tp_print */ From webhook-mailer at python.org Mon Sep 10 13:02:36 2018 From: webhook-mailer at python.org (Petr Viktorin) Date: Mon, 10 Sep 2018 17:02:36 -0000 Subject: [Python-checkins] bpo-26502: Implement FrameSummary.__len__() (GH-8632) Message-ID: https://github.com/python/cpython/commit/9797b7ae4496627836c55333765e10201a9840e3 commit: 9797b7ae4496627836c55333765e10201a9840e3 branch: master author: Berker Peksag committer: Petr Viktorin date: 2018-09-10T10:02:33-07:00 summary: bpo-26502: Implement FrameSummary.__len__() (GH-8632) files: A Misc/NEWS.d/next/Library/2018-08-02-20-39-32.bpo-26502.eGXr_k.rst M Lib/test/test_traceback.py M Lib/traceback.py diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index 8a3aa8a8648f..3af85b81721e 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -868,6 +868,7 @@ def extract(): (__file__, lineno+2, 'test_extract_stack', 'result = extract()'), (__file__, lineno+1, 'extract', 'return traceback.extract_stack()'), ]) + self.assertEqual(len(result[0]), 4) class TestFrame(unittest.TestCase): @@ -900,6 +901,10 @@ def test_explicit_line(self): f = traceback.FrameSummary("f", 1, "dummy", line="line") self.assertEqual("line", f.line) + def test_len(self): + f = traceback.FrameSummary("f", 1, "dummy", line="line") + self.assertEqual(len(f), 4) + class TestStack(unittest.TestCase): diff --git a/Lib/traceback.py b/Lib/traceback.py index 4e7605d15fa4..ab35da94b51e 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -279,6 +279,9 @@ def __repr__(self): return "".format( filename=self.filename, lineno=self.lineno, name=self.name) + def __len__(self): + return 4 + @property def line(self): if self._line is None: diff --git a/Misc/NEWS.d/next/Library/2018-08-02-20-39-32.bpo-26502.eGXr_k.rst b/Misc/NEWS.d/next/Library/2018-08-02-20-39-32.bpo-26502.eGXr_k.rst new file mode 100644 index 000000000000..3a3fc17e22df --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-08-02-20-39-32.bpo-26502.eGXr_k.rst @@ -0,0 +1,2 @@ +Implement ``traceback.FrameSummary.__len__()`` method to preserve +compatibility with the old tuple API. From webhook-mailer at python.org Mon Sep 10 13:23:01 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 17:23:01 -0000 Subject: [Python-checkins] [3.7] Fix misleading mentions of tp_size in comments. (GH-9136) Message-ID: https://github.com/python/cpython/commit/4d5d219a7ae1351e6f6e7c1c9845a499927d1219 commit: 4d5d219a7ae1351e6f6e7c1c9845a499927d1219 branch: 3.7 author: Benjamin Peterson committer: GitHub date: 2018-09-10T10:22:55-07:00 summary: [3.7] Fix misleading mentions of tp_size in comments. (GH-9136) Many type object initializations labeled a field "tp_size" in the comment, but the name of that field is tp_basicsize.. (cherry picked from commit 0e0bc4e221f592f305d335faf5f8046484eb9238) Co-authored-by: Peter Eisentraut files: M Modules/_abc.c M Modules/_blake2/blake2b_impl.c M Modules/_blake2/blake2s_impl.c M Modules/_queuemodule.c M Modules/_sha3/sha3module.c M Modules/_threadmodule.c M Modules/md5module.c M Modules/mmapmodule.c M Modules/ossaudiodev.c M Modules/sha1module.c M Modules/sha256module.c M Modules/sha512module.c M Objects/moduleobject.c M Objects/namespaceobject.c diff --git a/Modules/_abc.c b/Modules/_abc.c index ce9140fd03cc..9de199fa143f 100644 --- a/Modules/_abc.c +++ b/Modules/_abc.c @@ -68,7 +68,7 @@ PyDoc_STRVAR(abc_data_doc, static PyTypeObject _abc_data_type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_abc_data", /*tp_name*/ - sizeof(_abc_data), /*tp_size*/ + sizeof(_abc_data), /*tp_basicsize*/ .tp_dealloc = (destructor)abc_data_dealloc, .tp_flags = Py_TPFLAGS_DEFAULT, .tp_alloc = PyType_GenericAlloc, diff --git a/Modules/_blake2/blake2b_impl.c b/Modules/_blake2/blake2b_impl.c index f143b5d0c2ff..31eaf3e5a799 100644 --- a/Modules/_blake2/blake2b_impl.c +++ b/Modules/_blake2/blake2b_impl.c @@ -413,7 +413,7 @@ py_blake2b_dealloc(PyObject *self) PyTypeObject PyBlake2_BLAKE2bType = { PyVarObject_HEAD_INIT(NULL, 0) "_blake2.blake2b", /* tp_name */ - sizeof(BLAKE2bObject), /* tp_size */ + sizeof(BLAKE2bObject), /* tp_basicsize */ 0, /* tp_itemsize */ py_blake2b_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/_blake2/blake2s_impl.c b/Modules/_blake2/blake2s_impl.c index 7d366d394005..7c7cb4e35c7d 100644 --- a/Modules/_blake2/blake2s_impl.c +++ b/Modules/_blake2/blake2s_impl.c @@ -413,7 +413,7 @@ py_blake2s_dealloc(PyObject *self) PyTypeObject PyBlake2_BLAKE2sType = { PyVarObject_HEAD_INIT(NULL, 0) "_blake2.blake2s", /* tp_name */ - sizeof(BLAKE2sObject), /* tp_size */ + sizeof(BLAKE2sObject), /* tp_basicsize */ 0, /* tp_itemsize */ py_blake2s_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/_queuemodule.c b/Modules/_queuemodule.c index 0eb993027459..2a9406cc4760 100644 --- a/Modules/_queuemodule.c +++ b/Modules/_queuemodule.c @@ -309,7 +309,7 @@ static PyMethodDef simplequeue_methods[] = { static PyTypeObject PySimpleQueueType = { PyVarObject_HEAD_INIT(NULL, 0) "_queue.SimpleQueue", /*tp_name*/ - sizeof(simplequeueobject), /*tp_size*/ + sizeof(simplequeueobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)simplequeue_dealloc, /*tp_dealloc*/ diff --git a/Modules/_sha3/sha3module.c b/Modules/_sha3/sha3module.c index f73bc99eafe9..d879e9243c6e 100644 --- a/Modules/_sha3/sha3module.c +++ b/Modules/_sha3/sha3module.c @@ -489,7 +489,7 @@ static PyGetSetDef SHA3_getseters[] = { static PyTypeObject type_obj = { \ PyVarObject_HEAD_INIT(NULL, 0) \ type_name, /* tp_name */ \ - sizeof(SHA3object), /* tp_size */ \ + sizeof(SHA3object), /* tp_basicsize */ \ 0, /* tp_itemsize */ \ /* methods */ \ (destructor)SHA3_dealloc, /* tp_dealloc */ \ diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index f594bda678a8..16d3191c621a 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -226,7 +226,7 @@ static PyMethodDef lock_methods[] = { static PyTypeObject Locktype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_thread.lock", /*tp_name*/ - sizeof(lockobject), /*tp_size*/ + sizeof(lockobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)lock_dealloc, /*tp_dealloc*/ @@ -487,7 +487,7 @@ static PyMethodDef rlock_methods[] = { static PyTypeObject RLocktype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_thread.RLock", /*tp_name*/ - sizeof(rlockobject), /*tp_size*/ + sizeof(rlockobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)rlock_dealloc, /*tp_dealloc*/ diff --git a/Modules/md5module.c b/Modules/md5module.c index b019f8287684..c66b27308f05 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -465,7 +465,7 @@ static PyGetSetDef MD5_getseters[] = { static PyTypeObject MD5type = { PyVarObject_HEAD_INIT(NULL, 0) "_md5.md5", /*tp_name*/ - sizeof(MD5object), /*tp_size*/ + sizeof(MD5object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ MD5_dealloc, /*tp_dealloc*/ diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index 27030db49b24..ba42f7380c09 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -980,7 +980,7 @@ To map anonymous memory, pass -1 as the fileno (both versions)."); static PyTypeObject mmap_object_type = { PyVarObject_HEAD_INIT(NULL, 0) "mmap.mmap", /* tp_name */ - sizeof(mmap_object), /* tp_size */ + sizeof(mmap_object), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor) mmap_object_dealloc, /* tp_dealloc */ diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 58ee71f90108..150a14eb388a 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -964,7 +964,7 @@ oss_getattro(oss_audio_t *self, PyObject *nameobj) static PyTypeObject OSSAudioType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_audio_device", /*tp_name*/ - sizeof(oss_audio_t), /*tp_size*/ + sizeof(oss_audio_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_dealloc, /*tp_dealloc*/ @@ -996,7 +996,7 @@ static PyTypeObject OSSAudioType = { static PyTypeObject OSSMixerType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_mixer_device", /*tp_name*/ - sizeof(oss_mixer_t), /*tp_size*/ + sizeof(oss_mixer_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_mixer_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha1module.c b/Modules/sha1module.c index d39190b4d5e6..358cd1e7f9ac 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -442,7 +442,7 @@ static PyGetSetDef SHA1_getseters[] = { static PyTypeObject SHA1type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha1.sha1", /*tp_name*/ - sizeof(SHA1object), /*tp_size*/ + sizeof(SHA1object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA1_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha256module.c b/Modules/sha256module.c index e4cb3286ce95..3599eaf8be92 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -529,7 +529,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA224type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha224", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ @@ -563,7 +563,7 @@ static PyTypeObject SHA224type = { static PyTypeObject SHA256type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha256", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha512module.c b/Modules/sha512module.c index 5ac2a2a61cf5..d8c846a450f2 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -594,7 +594,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA384type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha384", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ @@ -628,7 +628,7 @@ static PyTypeObject SHA384type = { static PyTypeObject SHA512type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha512", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index 8fb368e41474..c4bf08141268 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -35,7 +35,7 @@ bad_traverse_test(PyObject *self, void *arg) { PyTypeObject PyModuleDef_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "moduledef", /* tp_name */ - sizeof(struct PyModuleDef), /* tp_size */ + sizeof(struct PyModuleDef), /* tp_basicsize */ 0, /* tp_itemsize */ }; @@ -789,7 +789,7 @@ static PyMethodDef module_methods[] = { PyTypeObject PyModule_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "module", /* tp_name */ - sizeof(PyModuleObject), /* tp_size */ + sizeof(PyModuleObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)module_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Objects/namespaceobject.c b/Objects/namespaceobject.c index e5698e6378de..7de102eb06a0 100644 --- a/Objects/namespaceobject.c +++ b/Objects/namespaceobject.c @@ -201,7 +201,7 @@ SimpleNamespace(**kwargs)"); PyTypeObject _PyNamespace_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "types.SimpleNamespace", /* tp_name */ - sizeof(_PyNamespaceObject), /* tp_size */ + sizeof(_PyNamespaceObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)namespace_dealloc, /* tp_dealloc */ 0, /* tp_print */ From webhook-mailer at python.org Mon Sep 10 13:23:07 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 17:23:07 -0000 Subject: [Python-checkins] [3.6] Fix misleading mentions of tp_size in comments (GH-9137) Message-ID: https://github.com/python/cpython/commit/6a939c8dae225aaf088c8c37de8cc2655c83ee61 commit: 6a939c8dae225aaf088c8c37de8cc2655c83ee61 branch: 3.6 author: Benjamin Peterson committer: GitHub date: 2018-09-10T10:23:03-07:00 summary: [3.6] Fix misleading mentions of tp_size in comments (GH-9137) Many type object initializations labeled a field "tp_size" in the comment, but the name of that field is tp_basicsize.. (cherry picked from commit 0e0bc4e221f592f305d335faf5f8046484eb9238) Co-authored-by: Peter Eisentraut files: M Modules/_blake2/blake2b_impl.c M Modules/_blake2/blake2s_impl.c M Modules/_sha3/sha3module.c M Modules/_threadmodule.c M Modules/md5module.c M Modules/mmapmodule.c M Modules/ossaudiodev.c M Modules/sha1module.c M Modules/sha256module.c M Modules/sha512module.c M Objects/moduleobject.c M Objects/namespaceobject.c diff --git a/Modules/_blake2/blake2b_impl.c b/Modules/_blake2/blake2b_impl.c index 09b200ebe172..9e6f72e67f49 100644 --- a/Modules/_blake2/blake2b_impl.c +++ b/Modules/_blake2/blake2b_impl.c @@ -423,7 +423,7 @@ py_blake2b_dealloc(PyObject *self) PyTypeObject PyBlake2_BLAKE2bType = { PyVarObject_HEAD_INIT(NULL, 0) "_blake2.blake2b", /* tp_name */ - sizeof(BLAKE2bObject), /* tp_size */ + sizeof(BLAKE2bObject), /* tp_basicsize */ 0, /* tp_itemsize */ py_blake2b_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/_blake2/blake2s_impl.c b/Modules/_blake2/blake2s_impl.c index 735b6f08deb6..0743fdf2084c 100644 --- a/Modules/_blake2/blake2s_impl.c +++ b/Modules/_blake2/blake2s_impl.c @@ -423,7 +423,7 @@ py_blake2s_dealloc(PyObject *self) PyTypeObject PyBlake2_BLAKE2sType = { PyVarObject_HEAD_INIT(NULL, 0) "_blake2.blake2s", /* tp_name */ - sizeof(BLAKE2sObject), /* tp_size */ + sizeof(BLAKE2sObject), /* tp_basicsize */ 0, /* tp_itemsize */ py_blake2s_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Modules/_sha3/sha3module.c b/Modules/_sha3/sha3module.c index 11aaa1f6b0af..e8504de16154 100644 --- a/Modules/_sha3/sha3module.c +++ b/Modules/_sha3/sha3module.c @@ -505,7 +505,7 @@ static PyGetSetDef SHA3_getseters[] = { static PyTypeObject type_obj = { \ PyVarObject_HEAD_INIT(NULL, 0) \ type_name, /* tp_name */ \ - sizeof(SHA3object), /* tp_size */ \ + sizeof(SHA3object), /* tp_basicsize */ \ 0, /* tp_itemsize */ \ /* methods */ \ (destructor)SHA3_dealloc, /* tp_dealloc */ \ diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 47e84b9e916d..c504b57b064f 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -234,7 +234,7 @@ static PyMethodDef lock_methods[] = { static PyTypeObject Locktype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_thread.lock", /*tp_name*/ - sizeof(lockobject), /*tp_size*/ + sizeof(lockobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)lock_dealloc, /*tp_dealloc*/ @@ -495,7 +495,7 @@ static PyMethodDef rlock_methods[] = { static PyTypeObject RLocktype = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "_thread.RLock", /*tp_name*/ - sizeof(rlockobject), /*tp_size*/ + sizeof(rlockobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)rlock_dealloc, /*tp_dealloc*/ diff --git a/Modules/md5module.c b/Modules/md5module.c index fe9da7d3af2d..01d2118339e3 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -466,7 +466,7 @@ static PyGetSetDef MD5_getseters[] = { static PyTypeObject MD5type = { PyVarObject_HEAD_INIT(NULL, 0) "_md5.md5", /*tp_name*/ - sizeof(MD5object), /*tp_size*/ + sizeof(MD5object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ MD5_dealloc, /*tp_dealloc*/ diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index ebef501d6749..721e5ed32ea2 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -1008,7 +1008,7 @@ To map anonymous memory, pass -1 as the fileno (both versions)."); static PyTypeObject mmap_object_type = { PyVarObject_HEAD_INIT(NULL, 0) "mmap.mmap", /* tp_name */ - sizeof(mmap_object), /* tp_size */ + sizeof(mmap_object), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor) mmap_object_dealloc, /* tp_dealloc */ diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 5f0505c8a7b4..da1d1631451f 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -969,7 +969,7 @@ oss_getattro(oss_audio_t *self, PyObject *nameobj) static PyTypeObject OSSAudioType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_audio_device", /*tp_name*/ - sizeof(oss_audio_t), /*tp_size*/ + sizeof(oss_audio_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_dealloc, /*tp_dealloc*/ @@ -1001,7 +1001,7 @@ static PyTypeObject OSSAudioType = { static PyTypeObject OSSMixerType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_mixer_device", /*tp_name*/ - sizeof(oss_mixer_t), /*tp_size*/ + sizeof(oss_mixer_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_mixer_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha1module.c b/Modules/sha1module.c index d5065ce13490..cd123586264c 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -443,7 +443,7 @@ static PyGetSetDef SHA1_getseters[] = { static PyTypeObject SHA1type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha1.sha1", /*tp_name*/ - sizeof(SHA1object), /*tp_size*/ + sizeof(SHA1object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA1_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha256module.c b/Modules/sha256module.c index 8f067f182c77..70c49b5da2ee 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -530,7 +530,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA224type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha224", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ @@ -564,7 +564,7 @@ static PyTypeObject SHA224type = { static PyTypeObject SHA256type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha256", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha512module.c b/Modules/sha512module.c index 17775ae57b6c..da0cb5775edb 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -599,7 +599,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA384type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha384", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ @@ -633,7 +633,7 @@ static PyTypeObject SHA384type = { static PyTypeObject SHA512type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha512", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index c5979329647b..0fba90c43c14 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -34,7 +34,7 @@ bad_traverse_test(PyObject *self, void *arg) { PyTypeObject PyModuleDef_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "moduledef", /* tp_name */ - sizeof(struct PyModuleDef), /* tp_size */ + sizeof(struct PyModuleDef), /* tp_basicsize */ 0, /* tp_itemsize */ }; @@ -761,7 +761,7 @@ The name must be a string; the optional doc argument can have any type."); PyTypeObject PyModule_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "module", /* tp_name */ - sizeof(PyModuleObject), /* tp_size */ + sizeof(PyModuleObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)module_dealloc, /* tp_dealloc */ 0, /* tp_print */ diff --git a/Objects/namespaceobject.c b/Objects/namespaceobject.c index d28c9133b4b7..6307ee0423dc 100644 --- a/Objects/namespaceobject.c +++ b/Objects/namespaceobject.c @@ -207,7 +207,7 @@ SimpleNamespace(**kwargs)"); PyTypeObject _PyNamespace_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "types.SimpleNamespace", /* tp_name */ - sizeof(_PyNamespaceObject), /* tp_size */ + sizeof(_PyNamespaceObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)namespace_dealloc, /* tp_dealloc */ 0, /* tp_print */ From webhook-mailer at python.org Mon Sep 10 13:29:46 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 17:29:46 -0000 Subject: [Python-checkins] closes bpo-34525: Fix smtplib's authobject() documentation (GH-8965) Message-ID: https://github.com/python/cpython/commit/78deb7f33227972987722bc3fed5bcb45fae869e commit: 78deb7f33227972987722bc3fed5bcb45fae869e branch: master author: Sebastian Rittau committer: Benjamin Peterson date: 2018-09-10T10:29:43-07:00 summary: closes bpo-34525: Fix smtplib's authobject() documentation (GH-8965) files: M Doc/library/smtplib.rst M Lib/smtplib.py diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 805217252ae8..6fb0934218a6 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -346,7 +346,7 @@ An :class:`SMTP` instance has the following methods: If optional keyword argument *initial_response_ok* is true, ``authobject()`` will be called first with no argument. It can return the - :rfc:`4954` "initial response" bytes which will be encoded and sent with + :rfc:`4954` "initial response" ASCII ``str`` which will be encoded and sent with the ``AUTH`` command as below. If the ``authobject()`` does not support an initial response (e.g. because it requires a challenge), it should return ``None`` when called with ``challenge=None``. If *initial_response_ok* is @@ -355,7 +355,7 @@ An :class:`SMTP` instance has the following methods: If the initial response check returns ``None``, or if *initial_response_ok* is false, ``authobject()`` will be called to process the server's challenge response; the *challenge* argument it is passed will be a ``bytes``. It - should return ``bytes`` *data* that will be base64 encoded and sent to the + should return ASCII ``str`` *data* that will be base64 encoded and sent to the server. The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, diff --git a/Lib/smtplib.py b/Lib/smtplib.py index 048c6bfb0671..5e1bc0b198ed 100755 --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -615,7 +615,7 @@ def auth(self, mechanism, authobject, *, initial_response_ok=True): It will be called to process the server's challenge response; the challenge argument it is passed will be a bytes. It should return - bytes data that will be base64 encoded and sent to the server. + an ASCII string that will be base64 encoded and sent to the server. Keyword arguments: - initial_response_ok: Allow sending the RFC 4954 initial-response From webhook-mailer at python.org Mon Sep 10 13:50:24 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 17:50:24 -0000 Subject: [Python-checkins] [2.7] Fix misleading mentions of tp_size in comments (GH-9138) Message-ID: https://github.com/python/cpython/commit/ed62e645101fdd83aee519f6f33a4384ff79f105 commit: ed62e645101fdd83aee519f6f33a4384ff79f105 branch: 2.7 author: Benjamin Peterson committer: GitHub date: 2018-09-10T10:50:15-07:00 summary: [2.7] Fix misleading mentions of tp_size in comments (GH-9138) Many type object initializations labeled a field "tp_size" in the comment, but the name of that field is tp_basicsize.. (cherry picked from commit 0e0bc4e221f592f305d335faf5f8046484eb9238) Co-authored-by: Peter Eisentraut files: M Modules/md5module.c M Modules/mmapmodule.c M Modules/ossaudiodev.c M Modules/sha256module.c M Modules/sha512module.c M Objects/moduleobject.c diff --git a/Modules/md5module.c b/Modules/md5module.c index 103da1497dac..dd90a22c00bf 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -236,7 +236,7 @@ copy() -- return a copy of the current md5 object"); static PyTypeObject MD5type = { PyVarObject_HEAD_INIT(NULL, 0) "_md5.md5", /*tp_name*/ - sizeof(md5object), /*tp_size*/ + sizeof(md5object), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)md5_dealloc, /*tp_dealloc*/ diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index bafdce3b1f84..02b31ca438a9 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -1026,7 +1026,7 @@ To map anonymous memory, pass -1 as the fileno (both versions)."); static PyTypeObject mmap_object_type = { PyVarObject_HEAD_INIT(NULL, 0) "mmap.mmap", /* tp_name */ - sizeof(mmap_object), /* tp_size */ + sizeof(mmap_object), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor) mmap_object_dealloc, /* tp_dealloc */ diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 284cc6107619..f833bb257f7a 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -842,7 +842,7 @@ oss_mixer_getattr(oss_mixer_t *self, char *name) static PyTypeObject OSSAudioType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_audio_device", /*tp_name*/ - sizeof(oss_audio_t), /*tp_size*/ + sizeof(oss_audio_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_dealloc, /*tp_dealloc*/ @@ -856,7 +856,7 @@ static PyTypeObject OSSAudioType = { static PyTypeObject OSSMixerType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "ossaudiodev.oss_mixer_device", /*tp_name*/ - sizeof(oss_mixer_t), /*tp_size*/ + sizeof(oss_mixer_t), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)oss_mixer_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha256module.c b/Modules/sha256module.c index 8f18faa17a59..76e52209f630 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -537,7 +537,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA224type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha224", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ @@ -571,7 +571,7 @@ static PyTypeObject SHA224type = { static PyTypeObject SHA256type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha256.sha256", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA_dealloc, /*tp_dealloc*/ diff --git a/Modules/sha512module.c b/Modules/sha512module.c index abc90218053f..e3733e0443ff 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -603,7 +603,7 @@ static PyMemberDef SHA_members[] = { static PyTypeObject SHA384type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha384", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ @@ -637,7 +637,7 @@ static PyTypeObject SHA384type = { static PyTypeObject SHA512type = { PyVarObject_HEAD_INIT(NULL, 0) "_sha512.sha512", /*tp_name*/ - sizeof(SHAobject), /*tp_size*/ + sizeof(SHAobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ SHA512_dealloc, /*tp_dealloc*/ diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index f2fed30e90dd..fd48ff176e75 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -221,7 +221,7 @@ The name must be a string; the optional doc argument can have any type."); PyTypeObject PyModule_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "module", /* tp_name */ - sizeof(PyModuleObject), /* tp_size */ + sizeof(PyModuleObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)module_dealloc, /* tp_dealloc */ 0, /* tp_print */ From webhook-mailer at python.org Mon Sep 10 14:10:05 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Mon, 10 Sep 2018 18:10:05 -0000 Subject: [Python-checkins] bpo-33604: Remove deprecated HMAC default value marked for removal in 3.8 (GH-7063) Message-ID: https://github.com/python/cpython/commit/51a4743d19abd016f0772a57fb31df7af9220e18 commit: 51a4743d19abd016f0772a57fb31df7af9220e18 branch: master author: Matthias Bussonnier committer: Gregory P. Smith date: 2018-09-10T11:10:01-07:00 summary: bpo-33604: Remove deprecated HMAC default value marked for removal in 3.8 (GH-7063) HMAC's digestmod was deprecated marked for removal, this removes it as planned. files: A Misc/NEWS.d/next/Library/2018-05-22-11-55-33.bpo-33604.6V4JcO.rst M Doc/library/hmac.rst M Lib/hmac.py M Lib/test/test_hmac.py diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index 731624ba94e3..dc994b07c35c 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -19,8 +19,7 @@ This module implements the HMAC algorithm as described by :rfc:`2104`. Return a new hmac object. *key* is a bytes or bytearray object giving the secret key. If *msg* is present, the method call ``update(msg)`` is made. *digestmod* is the digest name, digest constructor or module for the HMAC - object to use. It supports any name suitable to :func:`hashlib.new` and - defaults to the :data:`hashlib.md5` constructor. + object to use. It supports any name suitable to :func:`hashlib.new`. .. versionchanged:: 3.4 Parameter *key* can be a bytes or bytearray object. diff --git a/Lib/hmac.py b/Lib/hmac.py index 43b721297637..890eaba08e8f 100644 --- a/Lib/hmac.py +++ b/Lib/hmac.py @@ -35,12 +35,9 @@ def __init__(self, key, msg = None, digestmod = None): key: key for the keyed hash object. msg: Initial input for the hash, if provided. - digestmod: A module supporting PEP 247. *OR* - A hashlib constructor returning a new hash object. *OR* + digestmod: Required. A module supporting PEP 247. *OR* + A hashlib constructor returning a new hash object. *OR* A hash name suitable for hashlib.new(). - Defaults to hashlib.md5. - Implicit default to hashlib.md5 is deprecated since Python - 3.4 and will be removed in Python 3.8. Note: key and msg must be a bytes or bytearray objects. """ @@ -49,11 +46,7 @@ def __init__(self, key, msg = None, digestmod = None): raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__) if digestmod is None: - _warnings.warn("HMAC() without an explicit digestmod argument " - "is deprecated since Python 3.4, and will be removed " - "in 3.8", - DeprecationWarning, 2) - digestmod = _hashlib.md5 + raise ValueError('`digestmod` is required.') if callable(digestmod): self.digest_cons = digestmod diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py index 7f4901307630..896bbe9ab798 100644 --- a/Lib/test/test_hmac.py +++ b/Lib/test/test_hmac.py @@ -302,45 +302,38 @@ def digest(self): hmac.HMAC(b'a', b'b', digestmod=MockCrazyHash) self.fail('Expected warning about small block_size') - def test_with_digestmod_warning(self): - with self.assertWarns(DeprecationWarning): + def test_with_digestmod_no_default(self): + with self.assertRaises(ValueError): key = b"\x0b" * 16 data = b"Hi There" - digest = "9294727A3638BB1C13F48EF8158BFC9D" - h = hmac.HMAC(key, data) - self.assertEqual(h.hexdigest().upper(), digest) - + hmac.HMAC(key, data, digestmod=None) class ConstructorTestCase(unittest.TestCase): - @ignore_warning def test_normal(self): # Standard constructor call. failed = 0 try: - h = hmac.HMAC(b"key") + h = hmac.HMAC(b"key", digestmod='md5') except Exception: self.fail("Standard constructor call raised exception.") - @ignore_warning def test_with_str_key(self): # Pass a key of type str, which is an error, because it expects a key # of type bytes with self.assertRaises(TypeError): - h = hmac.HMAC("key") + h = hmac.HMAC("key", digestmod='md5') - @ignore_warning def test_dot_new_with_str_key(self): # Pass a key of type str, which is an error, because it expects a key # of type bytes with self.assertRaises(TypeError): - h = hmac.new("key") + h = hmac.new("key", digestmod='md5') - @ignore_warning def test_withtext(self): # Constructor call with text. try: - h = hmac.HMAC(b"key", b"hash this!") + h = hmac.HMAC(b"key", b"hash this!", digestmod='md5') except Exception: self.fail("Constructor call with text argument raised exception.") self.assertEqual(h.hexdigest(), '34325b639da4cfd95735b381e28cb864') @@ -369,13 +362,6 @@ def test_withmodule(self): class SanityTestCase(unittest.TestCase): - @ignore_warning - def test_default_is_md5(self): - # Testing if HMAC defaults to MD5 algorithm. - # NOTE: this whitebox test depends on the hmac class internals - h = hmac.HMAC(b"key") - self.assertEqual(h.digest_cons, hashlib.md5) - def test_exercise_all_methods(self): # Exercising all methods once. # This must not raise any exceptions diff --git a/Misc/NEWS.d/next/Library/2018-05-22-11-55-33.bpo-33604.6V4JcO.rst b/Misc/NEWS.d/next/Library/2018-05-22-11-55-33.bpo-33604.6V4JcO.rst new file mode 100644 index 000000000000..200a9d5f661a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-05-22-11-55-33.bpo-33604.6V4JcO.rst @@ -0,0 +1,2 @@ +Remove HMAC default to md5 marked for removal in 3.8 (removal originally +planned in 3.6, bump to 3.8 in gh-7062). From webhook-mailer at python.org Mon Sep 10 14:13:17 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 18:13:17 -0000 Subject: [Python-checkins] closes bpo-34525: Fix smtplib's authobject() documentation (GH-8965) Message-ID: https://github.com/python/cpython/commit/011141f312f11b35ac163d9d52fe48e4bb61a814 commit: 011141f312f11b35ac163d9d52fe48e4bb61a814 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T11:13:13-07:00 summary: closes bpo-34525: Fix smtplib's authobject() documentation (GH-8965) (cherry picked from commit 78deb7f33227972987722bc3fed5bcb45fae869e) Co-authored-by: Sebastian Rittau files: M Doc/library/smtplib.rst M Lib/smtplib.py diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 805217252ae8..6fb0934218a6 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -346,7 +346,7 @@ An :class:`SMTP` instance has the following methods: If optional keyword argument *initial_response_ok* is true, ``authobject()`` will be called first with no argument. It can return the - :rfc:`4954` "initial response" bytes which will be encoded and sent with + :rfc:`4954` "initial response" ASCII ``str`` which will be encoded and sent with the ``AUTH`` command as below. If the ``authobject()`` does not support an initial response (e.g. because it requires a challenge), it should return ``None`` when called with ``challenge=None``. If *initial_response_ok* is @@ -355,7 +355,7 @@ An :class:`SMTP` instance has the following methods: If the initial response check returns ``None``, or if *initial_response_ok* is false, ``authobject()`` will be called to process the server's challenge response; the *challenge* argument it is passed will be a ``bytes``. It - should return ``bytes`` *data* that will be base64 encoded and sent to the + should return ASCII ``str`` *data* that will be base64 encoded and sent to the server. The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, diff --git a/Lib/smtplib.py b/Lib/smtplib.py index 048c6bfb0671..5e1bc0b198ed 100755 --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -615,7 +615,7 @@ def auth(self, mechanism, authobject, *, initial_response_ok=True): It will be called to process the server's challenge response; the challenge argument it is passed will be a bytes. It should return - bytes data that will be base64 encoded and sent to the server. + an ASCII string that will be base64 encoded and sent to the server. Keyword arguments: - initial_response_ok: Allow sending the RFC 4954 initial-response From webhook-mailer at python.org Mon Sep 10 14:16:15 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Sep 2018 18:16:15 -0000 Subject: [Python-checkins] closes bpo-34525: Fix smtplib's authobject() documentation (GH-8965) Message-ID: https://github.com/python/cpython/commit/c0db8177dc9a7f79941551576e1faea9724e027d commit: c0db8177dc9a7f79941551576e1faea9724e027d branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T11:16:11-07:00 summary: closes bpo-34525: Fix smtplib's authobject() documentation (GH-8965) (cherry picked from commit 78deb7f33227972987722bc3fed5bcb45fae869e) Co-authored-by: Sebastian Rittau files: M Doc/library/smtplib.rst M Lib/smtplib.py diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 805217252ae8..6fb0934218a6 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -346,7 +346,7 @@ An :class:`SMTP` instance has the following methods: If optional keyword argument *initial_response_ok* is true, ``authobject()`` will be called first with no argument. It can return the - :rfc:`4954` "initial response" bytes which will be encoded and sent with + :rfc:`4954` "initial response" ASCII ``str`` which will be encoded and sent with the ``AUTH`` command as below. If the ``authobject()`` does not support an initial response (e.g. because it requires a challenge), it should return ``None`` when called with ``challenge=None``. If *initial_response_ok* is @@ -355,7 +355,7 @@ An :class:`SMTP` instance has the following methods: If the initial response check returns ``None``, or if *initial_response_ok* is false, ``authobject()`` will be called to process the server's challenge response; the *challenge* argument it is passed will be a ``bytes``. It - should return ``bytes`` *data* that will be base64 encoded and sent to the + should return ASCII ``str`` *data* that will be base64 encoded and sent to the server. The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, diff --git a/Lib/smtplib.py b/Lib/smtplib.py index 048c6bfb0671..5e1bc0b198ed 100755 --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -615,7 +615,7 @@ def auth(self, mechanism, authobject, *, initial_response_ok=True): It will be called to process the server's challenge response; the challenge argument it is passed will be a bytes. It should return - bytes data that will be base64 encoded and sent to the server. + an ASCII string that will be base64 encoded and sent to the server. Keyword arguments: - initial_response_ok: Allow sending the RFC 4954 initial-response From webhook-mailer at python.org Mon Sep 10 14:21:09 2018 From: webhook-mailer at python.org (Ethan Furman) Date: Mon, 10 Sep 2018 18:21:09 -0000 Subject: [Python-checkins] bpo-33217: Raise TypeError for non-Enum lookups in Enums (GH-6651) Message-ID: https://github.com/python/cpython/commit/9430652535f88125d8003f342a8884d34885d876 commit: 9430652535f88125d8003f342a8884d34885d876 branch: master author: Rahul Jha committer: Ethan Furman date: 2018-09-10T11:21:04-07:00 summary: bpo-33217: Raise TypeError for non-Enum lookups in Enums (GH-6651) * bpo-33217: Raise TypeError for non-Enum lookups in Enums files: A Misc/NEWS.d/next/Library/2018-04-30-13-29-47.bpo-33217.TENDzd.rst M Doc/library/enum.rst M Lib/enum.py M Lib/test/test_enum.py diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index dd794b42fcbd..3f17f9947641 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -976,7 +976,7 @@ Enum Classes The :class:`EnumMeta` metaclass is responsible for providing the :meth:`__contains__`, :meth:`__dir__`, :meth:`__iter__` and other methods that allow one to do things with an :class:`Enum` class that fail on a typical -class, such as `list(Color)` or `some_var in Color`. :class:`EnumMeta` is +class, such as `list(Color)` or `some_enum_var in Color`. :class:`EnumMeta` is responsible for ensuring that various other methods on the final :class:`Enum` class are correct (such as :meth:`__new__`, :meth:`__getnewargs__`, :meth:`__str__` and :meth:`__repr__`). diff --git a/Lib/enum.py b/Lib/enum.py index 04d8ec1fa872..9d1aef372c12 100644 --- a/Lib/enum.py +++ b/Lib/enum.py @@ -303,6 +303,10 @@ def __call__(cls, value, names=None, *, module=None, qualname=None, type=None, s return cls._create_(value, names, module=module, qualname=qualname, type=type, start=start) def __contains__(cls, member): + if not isinstance(member, Enum): + raise TypeError( + "unsupported operand type(s) for 'in': '%s' and '%s'" % ( + type(member).__qualname__, cls.__class__.__qualname__)) return isinstance(member, cls) and member._name_ in cls._member_map_ def __delattr__(cls, attr): @@ -705,7 +709,9 @@ def _create_pseudo_member_(cls, value): def __contains__(self, other): if not isinstance(other, self.__class__): - return NotImplemented + raise TypeError( + "unsupported operand type(s) for 'in': '%s' and '%s'" % ( + type(other).__qualname__, self.__class__.__qualname__)) return other._value_ & self._value_ == other._value_ def __repr__(self): diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py index 97559712b1dc..68483e654254 100644 --- a/Lib/test/test_enum.py +++ b/Lib/test/test_enum.py @@ -325,7 +325,10 @@ class IntLogic(int, Enum): def test_contains(self): Season = self.Season self.assertIn(Season.AUTUMN, Season) - self.assertNotIn(3, Season) + with self.assertRaises(TypeError): + 3 in Season + with self.assertRaises(TypeError): + 'AUTUMN' in Season val = Season(3) self.assertIn(val, Season) @@ -1752,6 +1755,13 @@ class Open(Flag): AC = 3 CE = 1<<19 + class Color(Flag): + BLACK = 0 + RED = 1 + GREEN = 2 + BLUE = 4 + PURPLE = RED|BLUE + def test_str(self): Perm = self.Perm self.assertEqual(str(Perm.R), 'Perm.R') @@ -1954,7 +1964,21 @@ def test_pickle(self): test_pickle_dump_load(self.assertIs, FlagStooges.CURLY|FlagStooges.MOE) test_pickle_dump_load(self.assertIs, FlagStooges) - def test_containment(self): + def test_contains(self): + Open = self.Open + Color = self.Color + self.assertFalse(Color.BLACK in Open) + self.assertFalse(Open.RO in Color) + with self.assertRaises(TypeError): + 'BLACK' in Color + with self.assertRaises(TypeError): + 'RO' in Open + with self.assertRaises(TypeError): + 1 in Color + with self.assertRaises(TypeError): + 1 in Open + + def test_member_contains(self): Perm = self.Perm R, W, X = Perm RW = R | W @@ -2072,6 +2096,13 @@ class Open(IntFlag): AC = 3 CE = 1<<19 + class Color(IntFlag): + BLACK = 0 + RED = 1 + GREEN = 2 + BLUE = 4 + PURPLE = RED|BLUE + def test_type(self): Perm = self.Perm Open = self.Open @@ -2340,7 +2371,23 @@ def test_programatic_function_from_empty_tuple(self): self.assertEqual(len(lst), len(Thing)) self.assertEqual(len(Thing), 0, Thing) - def test_containment(self): + def test_contains(self): + Open = self.Open + Color = self.Color + self.assertTrue(Color.GREEN in Color) + self.assertTrue(Open.RW in Open) + self.assertFalse(Color.GREEN in Open) + self.assertFalse(Open.RW in Color) + with self.assertRaises(TypeError): + 'GREEN' in Color + with self.assertRaises(TypeError): + 'RW' in Open + with self.assertRaises(TypeError): + 2 in Color + with self.assertRaises(TypeError): + 2 in Open + + def test_member_contains(self): Perm = self.Perm R, W, X = Perm RW = R | W @@ -2359,6 +2406,8 @@ def test_containment(self): self.assertFalse(R in WX) self.assertFalse(W in RX) self.assertFalse(X in RW) + with self.assertRaises(TypeError): + self.assertFalse('test' in RW) def test_bool(self): Perm = self.Perm diff --git a/Misc/NEWS.d/next/Library/2018-04-30-13-29-47.bpo-33217.TENDzd.rst b/Misc/NEWS.d/next/Library/2018-04-30-13-29-47.bpo-33217.TENDzd.rst new file mode 100644 index 000000000000..071f5f16bfb7 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-04-30-13-29-47.bpo-33217.TENDzd.rst @@ -0,0 +1,2 @@ +Raise :exc:`TypeError` when looking up non-Enum objects in Enum classes and +Enum members. From webhook-mailer at python.org Mon Sep 10 14:33:12 2018 From: webhook-mailer at python.org (Raymond Hettinger) Date: Mon, 10 Sep 2018 18:33:12 -0000 Subject: [Python-checkins] bpo-20180: itertools.groupby Argument Clinic conversion (GH-4170) Message-ID: https://github.com/python/cpython/commit/3286ce4adee85c5ce8ab3ee3089f3cd44a017fd7 commit: 3286ce4adee85c5ce8ab3ee3089f3cd44a017fd7 branch: master author: Tal Einat committer: Raymond Hettinger date: 2018-09-10T11:33:08-07:00 summary: bpo-20180: itertools.groupby Argument Clinic conversion (GH-4170) files: A Modules/clinic/itertoolsmodule.c.h M Modules/itertoolsmodule.c diff --git a/Modules/clinic/itertoolsmodule.c.h b/Modules/clinic/itertoolsmodule.c.h new file mode 100644 index 000000000000..68e67494bc93 --- /dev/null +++ b/Modules/clinic/itertoolsmodule.c.h @@ -0,0 +1,64 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(itertools_groupby__doc__, +"groupby(iterable, key=None)\n" +"--\n" +"\n" +"make an iterator that returns consecutive keys and groups from the iterable\n" +"\n" +" iterable\n" +" Elements to divide into groups according to the key function.\n" +" key\n" +" A function for computing the group category for each element.\n" +" If the key function is not specified or is None, the element itself\n" +" is used for grouping."); + +static PyObject * +itertools_groupby_impl(PyTypeObject *type, PyObject *it, PyObject *keyfunc); + +static PyObject * +itertools_groupby(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"iterable", "key", NULL}; + static _PyArg_Parser _parser = {"O|O:groupby", _keywords, 0}; + PyObject *it; + PyObject *keyfunc = Py_None; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &it, &keyfunc)) { + goto exit; + } + return_value = itertools_groupby_impl(type, it, keyfunc); + +exit: + return return_value; +} + +static PyObject * +itertools__grouper_impl(PyTypeObject *type, PyObject *parent, + PyObject *tgtkey); + +static PyObject * +itertools__grouper(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *parent; + PyObject *tgtkey; + + if ((type == &_grouper_type) && + !_PyArg_NoKeywords("_grouper", kwargs)) { + goto exit; + } + if (!PyArg_ParseTuple(args, "O!O:_grouper", + &groupby_type, &parent, &tgtkey)) { + goto exit; + } + return_value = itertools__grouper_impl(type, parent, tgtkey); + +exit: + return return_value; +} +/*[clinic end generated code: output=82e10c91569d2b95 input=a9049054013a1b77]*/ diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c index 8a36755bfa72..3ad7e5c80651 100644 --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -7,6 +7,17 @@ by Raymond D. Hettinger */ +/*[clinic input] +module itertools +class itertools.groupby "groupbyobject *" "&groupby_type" +class itertools._grouper "_grouperobject *" "&_grouper_type" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=9d506f5bb9177570]*/ + +static PyTypeObject groupby_type; +static PyTypeObject _grouper_type; +#include "clinic/itertoolsmodule.c.h" + /* groupby object ************************************************************/ @@ -20,19 +31,27 @@ typedef struct { const void *currgrouper; /* borrowed reference */ } groupbyobject; -static PyTypeObject groupby_type; static PyObject *_grouper_create(groupbyobject *, PyObject *); +/*[clinic input] + at classmethod +itertools.groupby.__new__ + + iterable as it: object + Elements to divide into groups according to the key function. + key as keyfunc: object = None + A function for computing the group category for each element. + If the key function is not specified or is None, the element itself + is used for grouping. + +make an iterator that returns consecutive keys and groups from the iterable +[clinic start generated code]*/ + static PyObject * -groupby_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_groupby_impl(PyTypeObject *type, PyObject *it, PyObject *keyfunc) +/*[clinic end generated code: output=cbb1ae3a90fd4141 input=6b3d123e87ff65a1]*/ { - static char *kwargs[] = {"iterable", "key", NULL}; groupbyobject *gbo; - PyObject *it, *keyfunc = Py_None; - - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O:groupby", kwargs, - &it, &keyfunc)) - return NULL; gbo = (groupbyobject *)type->tp_alloc(type, 0); if (gbo == NULL) @@ -186,11 +205,6 @@ static PyMethodDef groupby_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(groupby_doc, -"groupby(iterable, key=None) -> make an iterator that returns consecutive\n\ -keys and groups from the iterable. If the key function is not specified or\n\ -is None, the element itself is used for grouping.\n"); - static PyTypeObject groupby_type = { PyVarObject_HEAD_INIT(NULL, 0) "itertools.groupby", /* tp_name */ @@ -214,7 +228,7 @@ static PyTypeObject groupby_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - groupby_doc, /* tp_doc */ + itertools_groupby__doc__, /* tp_doc */ (traverseproc)groupby_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -231,7 +245,7 @@ static PyTypeObject groupby_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - groupby_new, /* tp_new */ + itertools_groupby, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -244,16 +258,20 @@ typedef struct { PyObject *tgtkey; } _grouperobject; -static PyTypeObject _grouper_type; +/*[clinic input] + at classmethod +itertools._grouper.__new__ + + parent: object(subclass_of='&groupby_type') + tgtkey: object + / +[clinic start generated code]*/ static PyObject * -_grouper_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools__grouper_impl(PyTypeObject *type, PyObject *parent, + PyObject *tgtkey) +/*[clinic end generated code: output=462efb1cdebb5914 input=dc180d7771fc8c59]*/ { - PyObject *parent, *tgtkey; - - if (!PyArg_ParseTuple(args, "O!O", &groupby_type, &parent, &tgtkey)) - return NULL; - return _grouper_create((groupbyobject*) parent, tgtkey); } @@ -374,7 +392,7 @@ static PyTypeObject _grouper_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - _grouper_new, /* tp_new */ + itertools__grouper, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; From webhook-mailer at python.org Mon Sep 10 14:46:19 2018 From: webhook-mailer at python.org (Petr Viktorin) Date: Mon, 10 Sep 2018 18:46:19 -0000 Subject: [Python-checkins] bpo-25083: Python can sometimes create incorrect .pyc files (GH-8449) Message-ID: https://github.com/python/cpython/commit/f64c813de84011a84ca21d75a294861a9cc2dfdc commit: f64c813de84011a84ca21d75a294861a9cc2dfdc branch: 2.7 author: tzickel committer: Petr Viktorin date: 2018-09-10T11:46:14-07:00 summary: bpo-25083: Python can sometimes create incorrect .pyc files (GH-8449) Python 2 never checked for I/O error when reading .py files and thus could mistake an I/O error for EOF and create incorrect .pyc files. This adds an check for this and aborts on an error. files: A Misc/NEWS.d/next/Core and Builtins/2018-07-25-22-47-19.bpo-25083.HT_hXh.rst M Include/errcode.h M Parser/tokenizer.c M Python/pythonrun.c diff --git a/Include/errcode.h b/Include/errcode.h index becec80c8acf..5c5a0f7fa346 100644 --- a/Include/errcode.h +++ b/Include/errcode.h @@ -29,6 +29,7 @@ extern "C" { #define E_EOFS 23 /* EOF in triple-quoted string */ #define E_EOLS 24 /* EOL in single-quoted string */ #define E_LINECONT 25 /* Unexpected characters after a line continuation */ +#define E_IO 26 /* I/O error */ #ifdef __cplusplus } diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-07-25-22-47-19.bpo-25083.HT_hXh.rst b/Misc/NEWS.d/next/Core and Builtins/2018-07-25-22-47-19.bpo-25083.HT_hXh.rst new file mode 100644 index 000000000000..0dc44c43b387 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-07-25-22-47-19.bpo-25083.HT_hXh.rst @@ -0,0 +1,2 @@ +Adding I/O error checking when reading .py files and aborting importing on +error. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 61bfb4e1b7af..c6e61df533e2 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1681,6 +1681,11 @@ int PyTokenizer_Get(struct tok_state *tok, char **p_start, char **p_end) { int result = tok_get(tok, p_start, p_end); + if (tok->fp && ferror(tok->fp)) { + clearerr(tok->fp); + result = ERRORTOKEN; + tok->done = E_IO; + } if (tok->decoding_erred) { result = ERRORTOKEN; tok->done = E_DECODE; diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 5707c9f52452..2c9f55fbd1df 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1654,6 +1654,9 @@ err_input(perrdetail *err) Py_XDECREF(tb); break; } + case E_IO: + msg = "I/O error while reading"; + break; case E_LINECONT: msg = "unexpected character after line continuation character"; break; From webhook-mailer at python.org Mon Sep 10 17:35:41 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Mon, 10 Sep 2018 21:35:41 -0000 Subject: [Python-checkins] switch descriptor howto to return value annotation (GH-7796) Message-ID: https://github.com/python/cpython/commit/28ea4c284724283265e95d1d1716c9f1dfc2d741 commit: 28ea4c284724283265e95d1d1716c9f1dfc2d741 branch: master author: NotAFile committer: Benjamin Peterson date: 2018-09-10T14:35:38-07:00 summary: switch descriptor howto to return value annotation (GH-7796) files: M Doc/howto/descriptor.rst diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index 6e4aa3e975f6..7b00d947f46b 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -48,11 +48,11 @@ a flexible set of new tools for everyday Python programs. Descriptor Protocol ------------------- -``descr.__get__(self, obj, type=None) --> value`` +``descr.__get__(self, obj, type=None) -> value`` -``descr.__set__(self, obj, value) --> None`` +``descr.__set__(self, obj, value) -> None`` -``descr.__delete__(self, obj) --> None`` +``descr.__delete__(self, obj) -> None`` That is all there is to it. Define any of these methods and an object is considered a descriptor and can override default behavior upon being looked up From webhook-mailer at python.org Mon Sep 10 19:16:12 2018 From: webhook-mailer at python.org (Zachary Ware) Date: Mon, 10 Sep 2018 23:16:12 -0000 Subject: [Python-checkins] bpo-8110: Refactor platform detection in subprocess (GH-9053) Message-ID: https://github.com/python/cpython/commit/880d42a3b247306f67837aa95e23f7c3471a30a3 commit: 880d42a3b247306f67837aa95e23f7c3471a30a3 branch: master author: Zachary Ware committer: GitHub date: 2018-09-10T16:16:08-07:00 summary: bpo-8110: Refactor platform detection in subprocess (GH-9053) Check for functionality via imports rather than checking sys.platform specifically for Windows files: A Misc/NEWS.d/next/Library/2018-09-03-23-54-35.bpo-8110.FExWI_.rst M Lib/subprocess.py diff --git a/Lib/subprocess.py b/Lib/subprocess.py index e070011d980e..1e04d5e57013 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -41,18 +41,56 @@ then returns a (exitcode, output) tuple """ -import sys -_mswindows = (sys.platform == "win32") - +import builtins +import errno import io import os import time import signal -import builtins +import sys +import threading import warnings -import errno from time import monotonic as _time + +__all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "getstatusoutput", + "getoutput", "check_output", "run", "CalledProcessError", "DEVNULL", + "SubprocessError", "TimeoutExpired", "CompletedProcess"] + # NOTE: We intentionally exclude list2cmdline as it is + # considered an internal implementation detail. issue10838. + +try: + import msvcrt + import _winapi + _mswindows = True +except ModuleNotFoundError: + _mswindows = False + import _posixsubprocess + import select + import selectors +else: + from _winapi import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP, + STD_INPUT_HANDLE, STD_OUTPUT_HANDLE, + STD_ERROR_HANDLE, SW_HIDE, + STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW, + ABOVE_NORMAL_PRIORITY_CLASS, BELOW_NORMAL_PRIORITY_CLASS, + HIGH_PRIORITY_CLASS, IDLE_PRIORITY_CLASS, + NORMAL_PRIORITY_CLASS, REALTIME_PRIORITY_CLASS, + CREATE_NO_WINDOW, DETACHED_PROCESS, + CREATE_DEFAULT_ERROR_MODE, CREATE_BREAKAWAY_FROM_JOB) + + __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP", + "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE", + "STD_ERROR_HANDLE", "SW_HIDE", + "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW", + "STARTUPINFO", + "ABOVE_NORMAL_PRIORITY_CLASS", "BELOW_NORMAL_PRIORITY_CLASS", + "HIGH_PRIORITY_CLASS", "IDLE_PRIORITY_CLASS", + "NORMAL_PRIORITY_CLASS", "REALTIME_PRIORITY_CLASS", + "CREATE_NO_WINDOW", "DETACHED_PROCESS", + "CREATE_DEFAULT_ERROR_MODE", "CREATE_BREAKAWAY_FROM_JOB"]) + + # Exception classes used by this module. class SubprocessError(Exception): pass @@ -123,9 +161,6 @@ def stdout(self, value): if _mswindows: - import threading - import msvcrt - import _winapi class STARTUPINFO: def __init__(self, *, dwFlags=0, hStdInput=None, hStdOutput=None, hStdError=None, wShowWindow=0, lpAttributeList=None): @@ -148,53 +183,6 @@ def copy(self): wShowWindow=self.wShowWindow, lpAttributeList=attr_list) -else: - import _posixsubprocess - import select - import selectors - import threading - - # When select or poll has indicated that the file is writable, - # we can write up to _PIPE_BUF bytes without risk of blocking. - # POSIX defines PIPE_BUF as >= 512. - _PIPE_BUF = getattr(select, 'PIPE_BUF', 512) - - # poll/select have the advantage of not requiring any extra file - # descriptor, contrarily to epoll/kqueue (also, they require a single - # syscall). - if hasattr(selectors, 'PollSelector'): - _PopenSelector = selectors.PollSelector - else: - _PopenSelector = selectors.SelectSelector - - -__all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "getstatusoutput", - "getoutput", "check_output", "run", "CalledProcessError", "DEVNULL", - "SubprocessError", "TimeoutExpired", "CompletedProcess"] - # NOTE: We intentionally exclude list2cmdline as it is - # considered an internal implementation detail. issue10838. - -if _mswindows: - from _winapi import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP, - STD_INPUT_HANDLE, STD_OUTPUT_HANDLE, - STD_ERROR_HANDLE, SW_HIDE, - STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW, - ABOVE_NORMAL_PRIORITY_CLASS, BELOW_NORMAL_PRIORITY_CLASS, - HIGH_PRIORITY_CLASS, IDLE_PRIORITY_CLASS, - NORMAL_PRIORITY_CLASS, REALTIME_PRIORITY_CLASS, - CREATE_NO_WINDOW, DETACHED_PROCESS, - CREATE_DEFAULT_ERROR_MODE, CREATE_BREAKAWAY_FROM_JOB) - - __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP", - "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE", - "STD_ERROR_HANDLE", "SW_HIDE", - "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW", - "STARTUPINFO", - "ABOVE_NORMAL_PRIORITY_CLASS", "BELOW_NORMAL_PRIORITY_CLASS", - "HIGH_PRIORITY_CLASS", "IDLE_PRIORITY_CLASS", - "NORMAL_PRIORITY_CLASS", "REALTIME_PRIORITY_CLASS", - "CREATE_NO_WINDOW", "DETACHED_PROCESS", - "CREATE_DEFAULT_ERROR_MODE", "CREATE_BREAKAWAY_FROM_JOB"]) class Handle(int): closed = False @@ -215,6 +203,19 @@ def __repr__(self): __del__ = Close __str__ = __repr__ +else: + # When select or poll has indicated that the file is writable, + # we can write up to _PIPE_BUF bytes without risk of blocking. + # POSIX defines PIPE_BUF as >= 512. + _PIPE_BUF = getattr(select, 'PIPE_BUF', 512) + + # poll/select have the advantage of not requiring any extra file + # descriptor, contrarily to epoll/kqueue (also, they require a single + # syscall). + if hasattr(selectors, 'PollSelector'): + _PopenSelector = selectors.PollSelector + else: + _PopenSelector = selectors.SelectSelector # This lists holds Popen instances for which the underlying process had not diff --git a/Misc/NEWS.d/next/Library/2018-09-03-23-54-35.bpo-8110.FExWI_.rst b/Misc/NEWS.d/next/Library/2018-09-03-23-54-35.bpo-8110.FExWI_.rst new file mode 100644 index 000000000000..c29ace1a0fc6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-03-23-54-35.bpo-8110.FExWI_.rst @@ -0,0 +1,2 @@ +Refactored :mod:`subprocess` to check for Windows-specific modules rather +than ``sys.platform == 'win32'``. From python-checkins at python.org Mon Sep 10 19:16:36 2018 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 10 Sep 2018 23:16:36 +0000 Subject: [Python-checkins] =?utf-8?q?test=3A_test_edit?= Message-ID: <20180910231636.1.082BA84C6077764B@mg.python.org> https://hg.python.org/test/rev/f2b4f6e298e4 changeset: 249:f2b4f6e298e4 user: Benjamin Peterson date: Mon Sep 10 16:16:33 2018 -0700 summary: test edit files: a | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/a b/a --- a/a +++ b/a @@ -10,4 +10,4 @@ snthdiueoa (obviously it should have gone this direction the first time) spam eggs 42 -hello, there +hello, there! -- Repository URL: https://hg.python.org/test From webhook-mailer at python.org Mon Sep 10 20:45:02 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 00:45:02 -0000 Subject: [Python-checkins] switch descriptor howto to return value annotation (GH-7796) Message-ID: https://github.com/python/cpython/commit/f3d00ae3be27ede2fc834d9d2b0028c9ad39f492 commit: f3d00ae3be27ede2fc834d9d2b0028c9ad39f492 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T17:44:58-07:00 summary: switch descriptor howto to return value annotation (GH-7796) (cherry picked from commit 28ea4c284724283265e95d1d1716c9f1dfc2d741) Co-authored-by: NotAFile files: M Doc/howto/descriptor.rst diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index 5e85a9aa6594..e5412583a674 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -48,11 +48,11 @@ a flexible set of new tools for everyday Python programs. Descriptor Protocol ------------------- -``descr.__get__(self, obj, type=None) --> value`` +``descr.__get__(self, obj, type=None) -> value`` -``descr.__set__(self, obj, value) --> None`` +``descr.__set__(self, obj, value) -> None`` -``descr.__delete__(self, obj) --> None`` +``descr.__delete__(self, obj) -> None`` That is all there is to it. Define any of these methods and an object is considered a descriptor and can override default behavior upon being looked up From webhook-mailer at python.org Mon Sep 10 20:46:26 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 00:46:26 -0000 Subject: [Python-checkins] bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242) Message-ID: https://github.com/python/cpython/commit/ce34410b8b67f49d8275c05d51b3ead50cf97f48 commit: ce34410b8b67f49d8275c05d51b3ead50cf97f48 branch: master author: Gregory P. Smith committer: GitHub date: 2018-09-10T17:46:22-07:00 summary: bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242) When subprocess.Popen() stdin= stdout= or stderr= handles are specified and appear in pass_fds=, don't close the original fds after dup'ing them. This implementation and unittest primarily came from @izbyshev (see the PR) See also https://github.com/izbyshev/cpython/commit/b89b52f28490b69142d5c061604b3a3989cec66c This also removes the old manual p2cread, c2pwrite, and errwrite closing logic as inheritable flags and _close_open_fds takes care of that properly today without special treatment. This code is within child_exec() where it is the only thread so there is no race condition between the dup and _Py_set_inheritable_async_safe call. files: A Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst M Lib/test/test_subprocess.py M Modules/_posixsubprocess.c diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 4719773b67b7..8419061b2a90 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -2529,6 +2529,36 @@ def test_pass_fds_inheritable(self): self.assertEqual(os.get_inheritable(inheritable), True) self.assertEqual(os.get_inheritable(non_inheritable), False) + + # bpo-32270: Ensure that descriptors specified in pass_fds + # are inherited even if they are used in redirections. + # Contributed by @izbyshev. + def test_pass_fds_redirected(self): + """Regression test for https://bugs.python.org/issue32270.""" + fd_status = support.findfile("fd_status.py", subdir="subprocessdata") + pass_fds = [] + for _ in range(2): + fd = os.open(os.devnull, os.O_RDWR) + self.addCleanup(os.close, fd) + pass_fds.append(fd) + + stdout_r, stdout_w = os.pipe() + self.addCleanup(os.close, stdout_r) + self.addCleanup(os.close, stdout_w) + pass_fds.insert(1, stdout_w) + + with subprocess.Popen([sys.executable, fd_status], + stdin=pass_fds[0], + stdout=pass_fds[1], + stderr=pass_fds[2], + close_fds=True, + pass_fds=pass_fds): + output = os.read(stdout_r, 1024) + fds = {int(num) for num in output.split(b',')} + + self.assertEqual(fds, {0, 1, 2} | frozenset(pass_fds), f"output={output!a}") + + def test_stdout_stdin_are_single_inout_fd(self): with io.open(os.devnull, "r+") as inout: p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"], diff --git a/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst b/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst new file mode 100644 index 000000000000..83f68624c1be --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst @@ -0,0 +1,2 @@ +The subprocess module no longer mistakenly closes redirected fds even when +they were in pass_fds when outside of the default {0, 1, 2} set. diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 0150fcb0970c..aeb10f9ecfe4 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -422,10 +422,20 @@ child_exec(char *const exec_array[], /* When duping fds, if there arises a situation where one of the fds is either 0, 1 or 2, it is possible that it is overwritten (#12607). */ - if (c2pwrite == 0) + if (c2pwrite == 0) { POSIX_CALL(c2pwrite = dup(c2pwrite)); - while (errwrite == 0 || errwrite == 1) + /* issue32270 */ + if (_Py_set_inheritable_async_safe(c2pwrite, 0, NULL) < 0) { + goto error; + } + } + while (errwrite == 0 || errwrite == 1) { POSIX_CALL(errwrite = dup(errwrite)); + /* issue32270 */ + if (_Py_set_inheritable_async_safe(errwrite, 0, NULL) < 0) { + goto error; + } + } /* Dup fds for child. dup2() removes the CLOEXEC flag but we must do it ourselves if dup2() @@ -451,14 +461,8 @@ child_exec(char *const exec_array[], else if (errwrite != -1) POSIX_CALL(dup2(errwrite, 2)); /* stderr */ - /* Close pipe fds. Make sure we don't close the same fd more than */ - /* once, or standard fds. */ - if (p2cread > 2) - POSIX_CALL(close(p2cread)); - if (c2pwrite > 2 && c2pwrite != p2cread) - POSIX_CALL(close(c2pwrite)); - if (errwrite != c2pwrite && errwrite != p2cread && errwrite > 2) - POSIX_CALL(close(errwrite)); + /* We no longer manually close p2cread, c2pwrite, and errwrite here as + * _close_open_fds takes care when it is not already non-inheritable. */ if (cwd) POSIX_CALL(chdir(cwd)); From webhook-mailer at python.org Mon Sep 10 20:54:42 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 00:54:42 -0000 Subject: [Python-checkins] Remove obsolete comment about latin-1 in `normalize_encoding` (GH-8739) Message-ID: https://github.com/python/cpython/commit/ed2e9ab804606019f167ef914bde43bc135c725f commit: ed2e9ab804606019f167ef914bde43bc135c725f branch: master author: Anthony Sottile committer: Gregory P. Smith date: 2018-09-10T17:54:37-07:00 summary: Remove obsolete comment about latin-1 in `normalize_encoding` (GH-8739) This docstring has drifted since python2: https://github.com/python/cpython/blob/ca079a3ea30098aff3197c559a0e32d42dda6d84/Lib/encodings/__init__.py#L68 files: M Lib/encodings/__init__.py diff --git a/Lib/encodings/__init__.py b/Lib/encodings/__init__.py index 025b7a8da3de..d737d5339dce 100644 --- a/Lib/encodings/__init__.py +++ b/Lib/encodings/__init__.py @@ -49,8 +49,7 @@ def normalize_encoding(encoding): collapsed and replaced with a single underscore, e.g. ' -;#' becomes '_'. Leading and trailing underscores are removed. - Note that encoding names should be ASCII only; if they do use - non-ASCII characters, these must be Latin-1 compatible. + Note that encoding names should be ASCII only. """ if isinstance(encoding, bytes): From webhook-mailer at python.org Mon Sep 10 21:04:38 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 01:04:38 -0000 Subject: [Python-checkins] Lib/test/support: fix typo in docstring (GH-8506) Message-ID: https://github.com/python/cpython/commit/e578fa162eef1cef27bc7dbf7ca46ab6dc52688a commit: e578fa162eef1cef27bc7dbf7ca46ab6dc52688a branch: master author: Daniel Hahler committer: Gregory P. Smith date: 2018-09-10T18:04:33-07:00 summary: Lib/test/support: fix typo in docstring (GH-8506) files: M Lib/test/support/__init__.py diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 13b60f7e8153..de997b253a69 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1955,8 +1955,8 @@ def set_match_tests(patterns): def match_test_regex(test_id): if regex_match(test_id): - # The regex matchs the whole identifier like - # 'test.test_os.FileTests.test_access' + # The regex matches the whole identifier, for example + # 'test.test_os.FileTests.test_access'. return True else: # Try to match parts of the test identifier. From webhook-mailer at python.org Mon Sep 10 21:07:23 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 01:07:23 -0000 Subject: [Python-checkins] Use bytes.hex instead of binascii.hexlify in pbkdf2_hmac example (GH-8420) Message-ID: https://github.com/python/cpython/commit/959625b5a56545f1908a7002fe11eb4f0411b780 commit: 959625b5a56545f1908a7002fe11eb4f0411b780 branch: master author: Ville Skytt? committer: Gregory P. Smith date: 2018-09-10T18:07:19-07:00 summary: Use bytes.hex instead of binascii.hexlify in pbkdf2_hmac example (GH-8420) files: M Doc/library/hashlib.rst diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index baf6b0af242c..4a8d7058c68f 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -244,10 +244,10 @@ include a `salt `_. *dklen* is the length of the derived key. If *dklen* is ``None`` then the digest size of the hash algorithm *hash_name* is used, e.g. 64 for SHA-512. - >>> import hashlib, binascii + >>> import hashlib >>> dk = hashlib.pbkdf2_hmac('sha256', b'password', b'salt', 100000) - >>> binascii.hexlify(dk) - b'0394a2ede332c9a13eb82e9b24631604c31df978b4e2f0fbd2c549944f9d79a5' + >>> dk.hex() + '0394a2ede332c9a13eb82e9b24631604c31df978b4e2f0fbd2c549944f9d79a5' .. versionadded:: 3.4 From webhook-mailer at python.org Mon Sep 10 21:13:11 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 01:13:11 -0000 Subject: [Python-checkins] bpo-33460: remove ellipsis that look like continuation prompts (GH-7851) Message-ID: https://github.com/python/cpython/commit/f019579828ed62653e2d41c95278308fa076ccaf commit: f019579828ed62653e2d41c95278308fa076ccaf branch: master author: Lew Kurtz <37632626+lew18 at users.noreply.github.com> committer: Gregory P. Smith date: 2018-09-10T18:13:08-07:00 summary: bpo-33460: remove ellipsis that look like continuation prompts (GH-7851) Remove ellipsis that look like continuation prompts, has a side benefit of putting rest of error message in proper text color. files: A Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst M Doc/tutorial/introduction.rst diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 22a209c10333..e68c9b10d03e 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -223,10 +223,14 @@ This only works with two literals though, not with variables or expressions:: >>> prefix = 'Py' >>> prefix 'thon' # can't concatenate a variable and a string literal - ... + File "", line 1 + prefix 'thon' + ^ SyntaxError: invalid syntax >>> ('un' * 3) 'ium' - ... + File "", line 1 + ('un' * 3) 'ium' + ^ SyntaxError: invalid syntax If you want to concatenate variables or a variable and a literal, use ``+``:: @@ -320,10 +324,12 @@ Python strings cannot be changed --- they are :term:`immutable`. Therefore, assigning to an indexed position in the string results in an error:: >>> word[0] = 'J' - ... + Traceback (most recent call last): + File "", line 1, in TypeError: 'str' object does not support item assignment >>> word[2:] = 'py' - ... + Traceback (most recent call last): + File "", line 1, in TypeError: 'str' object does not support item assignment If you need a different string, you should create a new one:: diff --git a/Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst b/Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst new file mode 100644 index 000000000000..6ee63b08f682 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst @@ -0,0 +1 @@ +replaced ellipsis with correct error codes in tutorial chapter 3. From webhook-mailer at python.org Mon Sep 10 21:15:59 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 01:15:59 -0000 Subject: [Python-checkins] bpo-33487: improve BZ2File Deprecation and documentation. (GH-6785) Message-ID: https://github.com/python/cpython/commit/ffa198c642f9c67b84ef192bf0f7016c4249e570 commit: ffa198c642f9c67b84ef192bf0f7016c4249e570 branch: master author: Matthias Bussonnier committer: Gregory P. Smith date: 2018-09-10T18:15:56-07:00 summary: bpo-33487: improve BZ2File Deprecation and documentation. (GH-6785) Emit warning when None passed explicitly, list Python version since deprecation in warning message and docs. files: A Misc/NEWS.d/next/Documentation/2018-05-13-14-44-30.bpo-33487.iLDzFb.rst M Doc/library/bz2.rst M Lib/bz2.py diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index d5f622515a40..946cc67dd301 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -81,7 +81,7 @@ All of the classes in this module may safely be accessed from multiple threads. If *filename* is a file object (rather than an actual file name), a mode of ``'w'`` does not truncate the file, and is instead equivalent to ``'a'``. - The *buffering* argument is ignored. Its use is deprecated. + The *buffering* argument is ignored. Its use is deprecated since Python 3.0. If *mode* is ``'w'`` or ``'a'``, *compresslevel* can be a number between ``1`` and ``9`` specifying the level of compression: ``1`` produces the @@ -109,6 +109,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionadded:: 3.3 + + .. deprecated:: 3.0 + The keyword argument *buffering* was deprecated and is now ignored. + .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added. diff --git a/Lib/bz2.py b/Lib/bz2.py index 3924aaed1678..3ab099147190 100644 --- a/Lib/bz2.py +++ b/Lib/bz2.py @@ -24,6 +24,8 @@ # Value 2 no longer used _MODE_WRITE = 3 +_sentinel = object() + class BZ2File(_compression.BaseStream): @@ -36,7 +38,7 @@ class BZ2File(_compression.BaseStream): returned as bytes, and data to be written should be given as bytes. """ - def __init__(self, filename, mode="r", buffering=None, compresslevel=9): + def __init__(self, filename, mode="r", buffering=_sentinel, compresslevel=9): """Open a bzip2-compressed file. If filename is a str, bytes, or PathLike object, it gives the @@ -47,7 +49,7 @@ def __init__(self, filename, mode="r", buffering=None, compresslevel=9): 'x' for creating exclusively, or 'a' for appending. These can equivalently be given as 'rb', 'wb', 'xb', and 'ab'. - buffering is ignored. Its use is deprecated. + buffering is ignored since Python 3.0. Its use is deprecated. If mode is 'w', 'x' or 'a', compresslevel can be a number between 1 and 9 specifying the level of compression: 1 produces the least @@ -63,9 +65,11 @@ def __init__(self, filename, mode="r", buffering=None, compresslevel=9): self._closefp = False self._mode = _MODE_CLOSED - if buffering is not None: - warnings.warn("Use of 'buffering' argument is deprecated", - DeprecationWarning) + if buffering is not _sentinel: + warnings.warn("Use of 'buffering' argument is deprecated and ignored" + "since Python 3.0.", + DeprecationWarning, + stacklevel=2) if not (1 <= compresslevel <= 9): raise ValueError("compresslevel must be between 1 and 9") diff --git a/Misc/NEWS.d/next/Documentation/2018-05-13-14-44-30.bpo-33487.iLDzFb.rst b/Misc/NEWS.d/next/Documentation/2018-05-13-14-44-30.bpo-33487.iLDzFb.rst new file mode 100644 index 000000000000..0439d983d1bc --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-05-13-14-44-30.bpo-33487.iLDzFb.rst @@ -0,0 +1,3 @@ +BZ2file now emit a DeprecationWarning when buffering=None is passed, the +deprecation message and documentation also now explicitely state it is +deprecated since 3.0. From webhook-mailer at python.org Mon Sep 10 21:43:11 2018 From: webhook-mailer at python.org (Raymond Hettinger) Date: Tue, 11 Sep 2018 01:43:11 -0000 Subject: [Python-checkins] Fix missing line from example shell session (GH-9143) Message-ID: https://github.com/python/cpython/commit/2064bb6d576ff7016d59318038779f428b0f0f3f commit: 2064bb6d576ff7016d59318038779f428b0f0f3f branch: master author: Raymond Hettinger committer: GitHub date: 2018-09-10T18:43:08-07:00 summary: Fix missing line from example shell session (GH-9143) files: M Doc/library/collections.rst diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 2a83d3037277..495cfc2c234f 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -198,6 +198,7 @@ updates keys found deeper in the chain:: >>> d['lion'] = 'orange' # update an existing key two levels down >>> d['snake'] = 'red' # new keys get added to the topmost dict >>> del d['elephant'] # remove an existing key one level down + >>> d # display result DeepChainMap({'zebra': 'black', 'snake': 'red'}, {}, {'lion': 'orange'}) From webhook-mailer at python.org Mon Sep 10 21:49:42 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 01:49:42 -0000 Subject: [Python-checkins] bpo-33460: remove ellipsis that look like continuation prompts (GH-7851) Message-ID: https://github.com/python/cpython/commit/037582eb7f4d61604ef2dcc72f896117588eeb3c commit: 037582eb7f4d61604ef2dcc72f896117588eeb3c branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T18:49:39-07:00 summary: bpo-33460: remove ellipsis that look like continuation prompts (GH-7851) Remove ellipsis that look like continuation prompts, has a side benefit of putting rest of error message in proper text color. (cherry picked from commit f019579828ed62653e2d41c95278308fa076ccaf) Co-authored-by: Lew Kurtz <37632626+lew18 at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst M Doc/tutorial/introduction.rst diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 22a209c10333..e68c9b10d03e 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -223,10 +223,14 @@ This only works with two literals though, not with variables or expressions:: >>> prefix = 'Py' >>> prefix 'thon' # can't concatenate a variable and a string literal - ... + File "", line 1 + prefix 'thon' + ^ SyntaxError: invalid syntax >>> ('un' * 3) 'ium' - ... + File "", line 1 + ('un' * 3) 'ium' + ^ SyntaxError: invalid syntax If you want to concatenate variables or a variable and a literal, use ``+``:: @@ -320,10 +324,12 @@ Python strings cannot be changed --- they are :term:`immutable`. Therefore, assigning to an indexed position in the string results in an error:: >>> word[0] = 'J' - ... + Traceback (most recent call last): + File "", line 1, in TypeError: 'str' object does not support item assignment >>> word[2:] = 'py' - ... + Traceback (most recent call last): + File "", line 1, in TypeError: 'str' object does not support item assignment If you need a different string, you should create a new one:: diff --git a/Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst b/Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst new file mode 100644 index 000000000000..6ee63b08f682 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-06-22-08-38-29.bpo-33460.kHt4D0.rst @@ -0,0 +1 @@ +replaced ellipsis with correct error codes in tutorial chapter 3. From webhook-mailer at python.org Mon Sep 10 22:13:05 2018 From: webhook-mailer at python.org (Raymond Hettinger) Date: Tue, 11 Sep 2018 02:13:05 -0000 Subject: [Python-checkins] Fix missing line from example shell session (GH-9143) (GH-9155) Message-ID: https://github.com/python/cpython/commit/6df2005b3999e6087dc524bfdf27b0dc5896d172 commit: 6df2005b3999e6087dc524bfdf27b0dc5896d172 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Raymond Hettinger date: 2018-09-10T19:13:02-07:00 summary: Fix missing line from example shell session (GH-9143) (GH-9155) (cherry picked from commit 2064bb6d576ff7016d59318038779f428b0f0f3f) Co-authored-by: Raymond Hettinger files: M Doc/library/collections.rst diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 2a83d3037277..495cfc2c234f 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -198,6 +198,7 @@ updates keys found deeper in the chain:: >>> d['lion'] = 'orange' # update an existing key two levels down >>> d['snake'] = 'red' # new keys get added to the topmost dict >>> del d['elephant'] # remove an existing key one level down + >>> d # display result DeepChainMap({'zebra': 'black', 'snake': 'red'}, {}, {'lion': 'orange'}) From webhook-mailer at python.org Mon Sep 10 22:13:26 2018 From: webhook-mailer at python.org (Raymond Hettinger) Date: Tue, 11 Sep 2018 02:13:26 -0000 Subject: [Python-checkins] Fix missing line from example shell session (GH-9143) (GH-9156) Message-ID: https://github.com/python/cpython/commit/3cb90d1a0b32dc293de8adb52b69cdc947a18c46 commit: 3cb90d1a0b32dc293de8adb52b69cdc947a18c46 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Raymond Hettinger date: 2018-09-10T19:13:23-07:00 summary: Fix missing line from example shell session (GH-9143) (GH-9156) (cherry picked from commit 2064bb6d576ff7016d59318038779f428b0f0f3f) Co-authored-by: Raymond Hettinger files: M Doc/library/collections.rst diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 82ba05737413..25687348d16d 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -198,6 +198,7 @@ updates keys found deeper in the chain:: >>> d['lion'] = 'orange' # update an existing key two levels down >>> d['snake'] = 'red' # new keys get added to the topmost dict >>> del d['elephant'] # remove an existing key one level down + >>> d # display result DeepChainMap({'zebra': 'black', 'snake': 'red'}, {}, {'lion': 'orange'}) From webhook-mailer at python.org Tue Sep 11 00:04:11 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 04:04:11 -0000 Subject: [Python-checkins] bpo-34625: Update vendorized expat version to 2.2.6. (GH-9150) Message-ID: https://github.com/python/cpython/commit/5033aa77aacaa5505636f150e8d54baac5bdca9c commit: 5033aa77aacaa5505636f150e8d54baac5bdca9c branch: master author: Benjamin Peterson committer: GitHub date: 2018-09-10T21:04:00-07:00 summary: bpo-34625: Update vendorized expat version to 2.2.6. (GH-9150) files: A Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst M Modules/expat/expat.h M Modules/expat/siphash.h M Modules/expat/xmlparse.c M Modules/expat/xmltok_impl.c diff --git a/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst new file mode 100644 index 000000000000..0747ec54470f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst @@ -0,0 +1 @@ +Update vendorized expat library version to 2.2.6. diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h index 1f608c02d6fd..174c3fafda3f 100644 --- a/Modules/expat/expat.h +++ b/Modules/expat/expat.h @@ -264,7 +264,7 @@ XML_ParserCreate_MM(const XML_Char *encoding, const XML_Char *namespaceSeparator); /* Prepare a parser object to be re-used. This is particularly - valuable when memory allocation overhead is disproportionatly high, + valuable when memory allocation overhead is disproportionately high, such as when a large number of small documnents need to be parsed. All handlers are cleared from the parser, except for the unknownEncodingHandler. The parser's external state is re-initialized @@ -1076,7 +1076,7 @@ XML_GetFeatureList(void); */ #define XML_MAJOR_VERSION 2 #define XML_MINOR_VERSION 2 -#define XML_MICRO_VERSION 5 +#define XML_MICRO_VERSION 6 #ifdef __cplusplus } diff --git a/Modules/expat/siphash.h b/Modules/expat/siphash.h index 581872df7b49..4d6786d7839a 100644 --- a/Modules/expat/siphash.h +++ b/Modules/expat/siphash.h @@ -11,6 +11,12 @@ * -------------------------------------------------------------------------- * HISTORY: * + * 2018-07-08 (Anton Maklakov) + * - Add "fall through" markers for GCC's -Wimplicit-fallthrough + * + * 2017-11-03 (Sebastian Pipping) + * - Hide sip_tobin and sip_binof unless SIPHASH_TOBIN macro is defined + * * 2017-07-25 (Vadim Zeitlin) * - Fix use of SIPHASH_MAIN macro * @@ -151,6 +157,8 @@ static struct sipkey *sip_tokey(struct sipkey *key, const void *src) { } /* sip_tokey() */ +#ifdef SIPHASH_TOBIN + #define sip_binof(v) sip_tobin((unsigned char[8]){ 0 }, (v)) static void *sip_tobin(void *dst, uint64_t u64) { @@ -158,6 +166,8 @@ static void *sip_tobin(void *dst, uint64_t u64) { return dst; } /* sip_tobin() */ +#endif /* SIPHASH_TOBIN */ + static void sip_round(struct siphash *H, const int rounds) { int i; @@ -231,12 +241,19 @@ static uint64_t sip24_final(struct siphash *H) { switch (left) { case 7: b |= (uint64_t)H->buf[6] << 48; + /* fall through */ case 6: b |= (uint64_t)H->buf[5] << 40; + /* fall through */ case 5: b |= (uint64_t)H->buf[4] << 32; + /* fall through */ case 4: b |= (uint64_t)H->buf[3] << 24; + /* fall through */ case 3: b |= (uint64_t)H->buf[2] << 16; + /* fall through */ case 2: b |= (uint64_t)H->buf[1] << 8; + /* fall through */ case 1: b |= (uint64_t)H->buf[0] << 0; + /* fall through */ case 0: break; } diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 90a237f30eb8..c4f3ffc215c9 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -1,4 +1,4 @@ -/* 4b74aa710b4ed5ce464b0ce544852cb47bf905c85a49c7bae2749f5885cb966d (2.2.5+) +/* 19ac4776051591216f1874e34ee99b6a43a3784c8bd7d70efeb9258dd22b906a (2.2.6+) __ __ _ ___\ \/ /_ __ __ _| |_ / _ \\ /| '_ \ / _` | __| @@ -161,6 +161,9 @@ typedef char ICHAR; /* Round up n to be a multiple of sz, where sz is a power of 2. */ #define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1)) +/* Do safe (NULL-aware) pointer arithmetic */ +#define EXPAT_SAFE_PTR_DIFF(p, q) (((p) && (q)) ? ((p) - (q)) : 0) + /* Handle the case where memmove() doesn't exist. */ #ifndef HAVE_MEMMOVE #ifdef HAVE_BCOPY @@ -1820,6 +1823,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -1969,6 +1973,7 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -2026,39 +2031,46 @@ XML_GetBuffer(XML_Parser parser, int len) default: ; } - if (len > parser->m_bufferLim - parser->m_bufferEnd) { + if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) { #ifdef XML_CONTEXT_BYTES int keep; #endif /* defined XML_CONTEXT_BYTES */ /* Do not invoke signed arithmetic overflow: */ - int neededSize = (int) ((unsigned)len + (unsigned)(parser->m_bufferEnd - parser->m_bufferPtr)); + int neededSize = (int) ((unsigned)len + + (unsigned)EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, + parser->m_bufferPtr)); if (neededSize < 0) { parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } #ifdef XML_CONTEXT_BYTES - keep = (int)(parser->m_bufferPtr - parser->m_buffer); + keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; neededSize += keep; #endif /* defined XML_CONTEXT_BYTES */ - if (neededSize <= parser->m_bufferLim - parser->m_buffer) { + if (neededSize <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) { #ifdef XML_CONTEXT_BYTES - if (keep < parser->m_bufferPtr - parser->m_buffer) { - int offset = (int)(parser->m_bufferPtr - parser->m_buffer) - keep; + if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) { + int offset = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer) - keep; + /* The buffer pointers cannot be NULL here; we have at least some bytes in the buffer */ memmove(parser->m_buffer, &parser->m_buffer[offset], parser->m_bufferEnd - parser->m_bufferPtr + keep); parser->m_bufferEnd -= offset; parser->m_bufferPtr -= offset; } #else - memmove(parser->m_buffer, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferPtr = parser->m_buffer; + if (parser->m_buffer && parser->m_bufferPtr) { + memmove(parser->m_buffer, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + parser->m_bufferPtr = parser->m_buffer; + } #endif /* not defined XML_CONTEXT_BYTES */ } else { char *newBuf; - int bufferSize = (int)(parser->m_bufferLim - parser->m_bufferPtr); + int bufferSize = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr); if (bufferSize == 0) bufferSize = INIT_BUFFER_SIZE; do { @@ -2077,25 +2089,34 @@ XML_GetBuffer(XML_Parser parser, int len) parser->m_bufferLim = newBuf + bufferSize; #ifdef XML_CONTEXT_BYTES if (parser->m_bufferPtr) { - int keep = (int)(parser->m_bufferPtr - parser->m_buffer); + int keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; - memcpy(newBuf, &parser->m_bufferPtr[-keep], parser->m_bufferEnd - parser->m_bufferPtr + keep); + memcpy(newBuf, &parser->m_bufferPtr[-keep], + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep); FREE(parser, parser->m_buffer); parser->m_buffer = newBuf; - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr) + keep; + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep; parser->m_bufferPtr = parser->m_buffer + keep; } else { - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; parser->m_bufferPtr = parser->m_buffer = newBuf; } #else if (parser->m_bufferPtr) { - memcpy(newBuf, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); + memcpy(newBuf, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); FREE(parser, parser->m_buffer); + parser->m_bufferEnd = newBuf + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + } + else { + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; } - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); parser->m_bufferPtr = parser->m_buffer = newBuf; #endif /* not defined XML_CONTEXT_BYTES */ } @@ -2908,9 +2929,11 @@ doContent(XML_Parser parser, poolClear(&parser->m_tempPool); freeBindings(parser, bindings); } - if ((parser->m_tagLevel == 0) && - !((parser->m_parsingStatus.parsing == XML_FINISHED) || (parser->m_parsingStatus.parsing == XML_SUSPENDED))) { - return epilogProcessor(parser, next, end, nextPtr); + if ((parser->m_tagLevel == 0) && (parser->m_parsingStatus.parsing != XML_FINISHED)) { + if (parser->m_parsingStatus.parsing == XML_SUSPENDED) + parser->m_processor = epilogProcessor; + else + return epilogProcessor(parser, next, end, nextPtr); } break; case XML_TOK_END_TAG: @@ -4746,8 +4769,8 @@ doProlog(XML_Parser parser, return XML_ERROR_NO_MEMORY; parser->m_declEntity->publicId = NULL; } - /* fall through */ #endif /* XML_DTD */ + /* fall through */ case XML_ROLE_ENTITY_SYSTEM_ID: if (dtd->keepProcessing && parser->m_declEntity) { parser->m_declEntity->systemId = poolStoreString(&dtd->pool, enc, @@ -6643,7 +6666,6 @@ hash(XML_Parser parser, KEY s) { struct siphash state; struct sipkey key; - (void)sip_tobin; (void)sip24_valid; copy_salt_to_sipkey(parser, &key); sip24_init(&state, &key); diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c index 0403dd3d09a0..4d9ae7dc3896 100644 --- a/Modules/expat/xmltok_impl.c +++ b/Modules/expat/xmltok_impl.c @@ -74,6 +74,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ case BT_DIGIT: \ @@ -102,6 +103,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ ptr += MINBPC(enc); \ @@ -602,7 +604,7 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end, return XML_TOK_INVALID; } } - /* fall through */ + /* fall through */ case BT_EQUALS: { int open; @@ -1442,6 +1444,7 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end, case BT_NMSTRT: if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f)) break; + /* fall through */ default: switch (BYTE_TO_ASCII(enc, ptr)) { case 0x24: /* $ */ @@ -1659,8 +1662,8 @@ PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1, { for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) { if (end1 - ptr1 < MINBPC(enc)) { - /* This line cannot be executed. THe incoming data has already - * been tokenized once, so imcomplete characters like this have + /* This line cannot be executed. The incoming data has already + * been tokenized once, so incomplete characters like this have * already been eliminated from the input. Retaining the * paranoia check is still valuable, however. */ From webhook-mailer at python.org Tue Sep 11 00:26:46 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 04:26:46 -0000 Subject: [Python-checkins] bpo-34625: Update vendorized expat version to 2.2.6. (GH-9150) Message-ID: https://github.com/python/cpython/commit/7a501def4fb98a1d6f15e4a5a006141d6027f948 commit: 7a501def4fb98a1d6f15e4a5a006141d6027f948 branch: 2.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T21:26:42-07:00 summary: bpo-34625: Update vendorized expat version to 2.2.6. (GH-9150) (cherry picked from commit 5033aa77aacaa5505636f150e8d54baac5bdca9c) Co-authored-by: Benjamin Peterson files: A Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst M Modules/expat/expat.h M Modules/expat/siphash.h M Modules/expat/xmlparse.c M Modules/expat/xmltok_impl.c diff --git a/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst new file mode 100644 index 000000000000..0747ec54470f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst @@ -0,0 +1 @@ +Update vendorized expat library version to 2.2.6. diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h index 1f608c02d6fd..174c3fafda3f 100644 --- a/Modules/expat/expat.h +++ b/Modules/expat/expat.h @@ -264,7 +264,7 @@ XML_ParserCreate_MM(const XML_Char *encoding, const XML_Char *namespaceSeparator); /* Prepare a parser object to be re-used. This is particularly - valuable when memory allocation overhead is disproportionatly high, + valuable when memory allocation overhead is disproportionately high, such as when a large number of small documnents need to be parsed. All handlers are cleared from the parser, except for the unknownEncodingHandler. The parser's external state is re-initialized @@ -1076,7 +1076,7 @@ XML_GetFeatureList(void); */ #define XML_MAJOR_VERSION 2 #define XML_MINOR_VERSION 2 -#define XML_MICRO_VERSION 5 +#define XML_MICRO_VERSION 6 #ifdef __cplusplus } diff --git a/Modules/expat/siphash.h b/Modules/expat/siphash.h index 581872df7b49..4d6786d7839a 100644 --- a/Modules/expat/siphash.h +++ b/Modules/expat/siphash.h @@ -11,6 +11,12 @@ * -------------------------------------------------------------------------- * HISTORY: * + * 2018-07-08 (Anton Maklakov) + * - Add "fall through" markers for GCC's -Wimplicit-fallthrough + * + * 2017-11-03 (Sebastian Pipping) + * - Hide sip_tobin and sip_binof unless SIPHASH_TOBIN macro is defined + * * 2017-07-25 (Vadim Zeitlin) * - Fix use of SIPHASH_MAIN macro * @@ -151,6 +157,8 @@ static struct sipkey *sip_tokey(struct sipkey *key, const void *src) { } /* sip_tokey() */ +#ifdef SIPHASH_TOBIN + #define sip_binof(v) sip_tobin((unsigned char[8]){ 0 }, (v)) static void *sip_tobin(void *dst, uint64_t u64) { @@ -158,6 +166,8 @@ static void *sip_tobin(void *dst, uint64_t u64) { return dst; } /* sip_tobin() */ +#endif /* SIPHASH_TOBIN */ + static void sip_round(struct siphash *H, const int rounds) { int i; @@ -231,12 +241,19 @@ static uint64_t sip24_final(struct siphash *H) { switch (left) { case 7: b |= (uint64_t)H->buf[6] << 48; + /* fall through */ case 6: b |= (uint64_t)H->buf[5] << 40; + /* fall through */ case 5: b |= (uint64_t)H->buf[4] << 32; + /* fall through */ case 4: b |= (uint64_t)H->buf[3] << 24; + /* fall through */ case 3: b |= (uint64_t)H->buf[2] << 16; + /* fall through */ case 2: b |= (uint64_t)H->buf[1] << 8; + /* fall through */ case 1: b |= (uint64_t)H->buf[0] << 0; + /* fall through */ case 0: break; } diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 90a237f30eb8..c4f3ffc215c9 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -1,4 +1,4 @@ -/* 4b74aa710b4ed5ce464b0ce544852cb47bf905c85a49c7bae2749f5885cb966d (2.2.5+) +/* 19ac4776051591216f1874e34ee99b6a43a3784c8bd7d70efeb9258dd22b906a (2.2.6+) __ __ _ ___\ \/ /_ __ __ _| |_ / _ \\ /| '_ \ / _` | __| @@ -161,6 +161,9 @@ typedef char ICHAR; /* Round up n to be a multiple of sz, where sz is a power of 2. */ #define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1)) +/* Do safe (NULL-aware) pointer arithmetic */ +#define EXPAT_SAFE_PTR_DIFF(p, q) (((p) && (q)) ? ((p) - (q)) : 0) + /* Handle the case where memmove() doesn't exist. */ #ifndef HAVE_MEMMOVE #ifdef HAVE_BCOPY @@ -1820,6 +1823,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -1969,6 +1973,7 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -2026,39 +2031,46 @@ XML_GetBuffer(XML_Parser parser, int len) default: ; } - if (len > parser->m_bufferLim - parser->m_bufferEnd) { + if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) { #ifdef XML_CONTEXT_BYTES int keep; #endif /* defined XML_CONTEXT_BYTES */ /* Do not invoke signed arithmetic overflow: */ - int neededSize = (int) ((unsigned)len + (unsigned)(parser->m_bufferEnd - parser->m_bufferPtr)); + int neededSize = (int) ((unsigned)len + + (unsigned)EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, + parser->m_bufferPtr)); if (neededSize < 0) { parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } #ifdef XML_CONTEXT_BYTES - keep = (int)(parser->m_bufferPtr - parser->m_buffer); + keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; neededSize += keep; #endif /* defined XML_CONTEXT_BYTES */ - if (neededSize <= parser->m_bufferLim - parser->m_buffer) { + if (neededSize <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) { #ifdef XML_CONTEXT_BYTES - if (keep < parser->m_bufferPtr - parser->m_buffer) { - int offset = (int)(parser->m_bufferPtr - parser->m_buffer) - keep; + if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) { + int offset = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer) - keep; + /* The buffer pointers cannot be NULL here; we have at least some bytes in the buffer */ memmove(parser->m_buffer, &parser->m_buffer[offset], parser->m_bufferEnd - parser->m_bufferPtr + keep); parser->m_bufferEnd -= offset; parser->m_bufferPtr -= offset; } #else - memmove(parser->m_buffer, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferPtr = parser->m_buffer; + if (parser->m_buffer && parser->m_bufferPtr) { + memmove(parser->m_buffer, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + parser->m_bufferPtr = parser->m_buffer; + } #endif /* not defined XML_CONTEXT_BYTES */ } else { char *newBuf; - int bufferSize = (int)(parser->m_bufferLim - parser->m_bufferPtr); + int bufferSize = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr); if (bufferSize == 0) bufferSize = INIT_BUFFER_SIZE; do { @@ -2077,25 +2089,34 @@ XML_GetBuffer(XML_Parser parser, int len) parser->m_bufferLim = newBuf + bufferSize; #ifdef XML_CONTEXT_BYTES if (parser->m_bufferPtr) { - int keep = (int)(parser->m_bufferPtr - parser->m_buffer); + int keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; - memcpy(newBuf, &parser->m_bufferPtr[-keep], parser->m_bufferEnd - parser->m_bufferPtr + keep); + memcpy(newBuf, &parser->m_bufferPtr[-keep], + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep); FREE(parser, parser->m_buffer); parser->m_buffer = newBuf; - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr) + keep; + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep; parser->m_bufferPtr = parser->m_buffer + keep; } else { - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; parser->m_bufferPtr = parser->m_buffer = newBuf; } #else if (parser->m_bufferPtr) { - memcpy(newBuf, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); + memcpy(newBuf, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); FREE(parser, parser->m_buffer); + parser->m_bufferEnd = newBuf + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + } + else { + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; } - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); parser->m_bufferPtr = parser->m_buffer = newBuf; #endif /* not defined XML_CONTEXT_BYTES */ } @@ -2908,9 +2929,11 @@ doContent(XML_Parser parser, poolClear(&parser->m_tempPool); freeBindings(parser, bindings); } - if ((parser->m_tagLevel == 0) && - !((parser->m_parsingStatus.parsing == XML_FINISHED) || (parser->m_parsingStatus.parsing == XML_SUSPENDED))) { - return epilogProcessor(parser, next, end, nextPtr); + if ((parser->m_tagLevel == 0) && (parser->m_parsingStatus.parsing != XML_FINISHED)) { + if (parser->m_parsingStatus.parsing == XML_SUSPENDED) + parser->m_processor = epilogProcessor; + else + return epilogProcessor(parser, next, end, nextPtr); } break; case XML_TOK_END_TAG: @@ -4746,8 +4769,8 @@ doProlog(XML_Parser parser, return XML_ERROR_NO_MEMORY; parser->m_declEntity->publicId = NULL; } - /* fall through */ #endif /* XML_DTD */ + /* fall through */ case XML_ROLE_ENTITY_SYSTEM_ID: if (dtd->keepProcessing && parser->m_declEntity) { parser->m_declEntity->systemId = poolStoreString(&dtd->pool, enc, @@ -6643,7 +6666,6 @@ hash(XML_Parser parser, KEY s) { struct siphash state; struct sipkey key; - (void)sip_tobin; (void)sip24_valid; copy_salt_to_sipkey(parser, &key); sip24_init(&state, &key); diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c index 0403dd3d09a0..4d9ae7dc3896 100644 --- a/Modules/expat/xmltok_impl.c +++ b/Modules/expat/xmltok_impl.c @@ -74,6 +74,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ case BT_DIGIT: \ @@ -102,6 +103,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ ptr += MINBPC(enc); \ @@ -602,7 +604,7 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end, return XML_TOK_INVALID; } } - /* fall through */ + /* fall through */ case BT_EQUALS: { int open; @@ -1442,6 +1444,7 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end, case BT_NMSTRT: if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f)) break; + /* fall through */ default: switch (BYTE_TO_ASCII(enc, ptr)) { case 0x24: /* $ */ @@ -1659,8 +1662,8 @@ PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1, { for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) { if (end1 - ptr1 < MINBPC(enc)) { - /* This line cannot be executed. THe incoming data has already - * been tokenized once, so imcomplete characters like this have + /* This line cannot be executed. The incoming data has already + * been tokenized once, so incomplete characters like this have * already been eliminated from the input. Retaining the * paranoia check is still valuable, however. */ From webhook-mailer at python.org Tue Sep 11 00:36:24 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 04:36:24 -0000 Subject: [Python-checkins] bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242) (GH-9148) Message-ID: https://github.com/python/cpython/commit/9c4a63fc17efea31ad41f90d28825be37469e0e2 commit: 9c4a63fc17efea31ad41f90d28825be37469e0e2 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Gregory P. Smith date: 2018-09-10T21:36:20-07:00 summary: bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242) (GH-9148) When subprocess.Popen() stdin= stdout= or stderr= handles are specified and appear in pass_fds=, don't close the original fds after dup'ing them. This implementation and unittest primarily came from @izbyshev (see the PR) See also https://github.com/izbyshev/cpython/commit/b89b52f28490b69142d5c061604b3a3989cec66c This also removes the old manual p2cread, c2pwrite, and errwrite closing logic as inheritable flags and _close_open_fds takes care of that properly today without special treatment. This code is within child_exec() where it is the only thread so there is no race condition between the dup and _Py_set_inheritable_async_safe call. (cherry picked from commit ce34410b8b67f49d8275c05d51b3ead50cf97f48) Co-authored-by: Gregory P. Smith [Google] files: A Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst M Lib/test/test_subprocess.py M Modules/_posixsubprocess.c diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 4719773b67b7..8419061b2a90 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -2529,6 +2529,36 @@ def test_pass_fds_inheritable(self): self.assertEqual(os.get_inheritable(inheritable), True) self.assertEqual(os.get_inheritable(non_inheritable), False) + + # bpo-32270: Ensure that descriptors specified in pass_fds + # are inherited even if they are used in redirections. + # Contributed by @izbyshev. + def test_pass_fds_redirected(self): + """Regression test for https://bugs.python.org/issue32270.""" + fd_status = support.findfile("fd_status.py", subdir="subprocessdata") + pass_fds = [] + for _ in range(2): + fd = os.open(os.devnull, os.O_RDWR) + self.addCleanup(os.close, fd) + pass_fds.append(fd) + + stdout_r, stdout_w = os.pipe() + self.addCleanup(os.close, stdout_r) + self.addCleanup(os.close, stdout_w) + pass_fds.insert(1, stdout_w) + + with subprocess.Popen([sys.executable, fd_status], + stdin=pass_fds[0], + stdout=pass_fds[1], + stderr=pass_fds[2], + close_fds=True, + pass_fds=pass_fds): + output = os.read(stdout_r, 1024) + fds = {int(num) for num in output.split(b',')} + + self.assertEqual(fds, {0, 1, 2} | frozenset(pass_fds), f"output={output!a}") + + def test_stdout_stdin_are_single_inout_fd(self): with io.open(os.devnull, "r+") as inout: p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"], diff --git a/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst b/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst new file mode 100644 index 000000000000..83f68624c1be --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst @@ -0,0 +1,2 @@ +The subprocess module no longer mistakenly closes redirected fds even when +they were in pass_fds when outside of the default {0, 1, 2} set. diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 0150fcb0970c..aeb10f9ecfe4 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -422,10 +422,20 @@ child_exec(char *const exec_array[], /* When duping fds, if there arises a situation where one of the fds is either 0, 1 or 2, it is possible that it is overwritten (#12607). */ - if (c2pwrite == 0) + if (c2pwrite == 0) { POSIX_CALL(c2pwrite = dup(c2pwrite)); - while (errwrite == 0 || errwrite == 1) + /* issue32270 */ + if (_Py_set_inheritable_async_safe(c2pwrite, 0, NULL) < 0) { + goto error; + } + } + while (errwrite == 0 || errwrite == 1) { POSIX_CALL(errwrite = dup(errwrite)); + /* issue32270 */ + if (_Py_set_inheritable_async_safe(errwrite, 0, NULL) < 0) { + goto error; + } + } /* Dup fds for child. dup2() removes the CLOEXEC flag but we must do it ourselves if dup2() @@ -451,14 +461,8 @@ child_exec(char *const exec_array[], else if (errwrite != -1) POSIX_CALL(dup2(errwrite, 2)); /* stderr */ - /* Close pipe fds. Make sure we don't close the same fd more than */ - /* once, or standard fds. */ - if (p2cread > 2) - POSIX_CALL(close(p2cread)); - if (c2pwrite > 2 && c2pwrite != p2cread) - POSIX_CALL(close(c2pwrite)); - if (errwrite != c2pwrite && errwrite != p2cread && errwrite > 2) - POSIX_CALL(close(errwrite)); + /* We no longer manually close p2cread, c2pwrite, and errwrite here as + * _close_open_fds takes care when it is not already non-inheritable. */ if (cwd) POSIX_CALL(chdir(cwd)); From webhook-mailer at python.org Tue Sep 11 00:37:36 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 04:37:36 -0000 Subject: [Python-checkins] bpo-33032: Mention the implicit cache in struct.Struct() docs (GH-7700) Message-ID: https://github.com/python/cpython/commit/3666b3c1f695a145adab1bf644c22e564e8eb0ee commit: 3666b3c1f695a145adab1bf644c22e564e8eb0ee branch: master author: Zackery Spytz committer: Gregory P. Smith date: 2018-09-10T21:37:33-07:00 summary: bpo-33032: Mention the implicit cache in struct.Struct() docs (GH-7700) Mention the implicit cache in struct.Struct() docs. Consistent with the re.compile documentation note. files: M Doc/library/struct.rst diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index d6a3cb721e83..bad33ac666ed 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -405,6 +405,12 @@ The :mod:`struct` module also defines the following type: methods is more efficient than calling the :mod:`struct` functions with the same format since the format string only needs to be compiled once. + .. note:: + + The compiled versions of the most recent format strings passed to + :class:`Struct` and the module-level functions are cached, so programs + that use only a few format strings needn't worry about reusing a single + :class:`Struct` instance. Compiled Struct objects support the following methods and attributes: From webhook-mailer at python.org Tue Sep 11 00:50:46 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 04:50:46 -0000 Subject: [Python-checkins] bpo-33032: Mention the implicit cache in struct.Struct() docs (GH-7700) Message-ID: https://github.com/python/cpython/commit/2110f78d92522499836a928d268704fdc5f7a037 commit: 2110f78d92522499836a928d268704fdc5f7a037 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T21:50:41-07:00 summary: bpo-33032: Mention the implicit cache in struct.Struct() docs (GH-7700) Mention the implicit cache in struct.Struct() docs. Consistent with the re.compile documentation note. (cherry picked from commit 3666b3c1f695a145adab1bf644c22e564e8eb0ee) Co-authored-by: Zackery Spytz files: M Doc/library/struct.rst diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index 2d0866c7e09e..f10fbe4fc0df 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -405,6 +405,12 @@ The :mod:`struct` module also defines the following type: methods is more efficient than calling the :mod:`struct` functions with the same format since the format string only needs to be compiled once. + .. note:: + + The compiled versions of the most recent format strings passed to + :class:`Struct` and the module-level functions are cached, so programs + that use only a few format strings needn't worry about reusing a single + :class:`Struct` instance. Compiled Struct objects support the following methods and attributes: From webhook-mailer at python.org Tue Sep 11 00:55:30 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 04:55:30 -0000 Subject: [Python-checkins] bpo-34625: Update vendorized expat version to 2.2.6. (GH-9150) Message-ID: https://github.com/python/cpython/commit/2bc4eea8f6675b157adcaea4a59d982d58a5e7ae commit: 2bc4eea8f6675b157adcaea4a59d982d58a5e7ae branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T21:55:27-07:00 summary: bpo-34625: Update vendorized expat version to 2.2.6. (GH-9150) (cherry picked from commit 5033aa77aacaa5505636f150e8d54baac5bdca9c) Co-authored-by: Benjamin Peterson files: A Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst M Modules/expat/expat.h M Modules/expat/siphash.h M Modules/expat/xmlparse.c M Modules/expat/xmltok_impl.c diff --git a/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst new file mode 100644 index 000000000000..0747ec54470f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst @@ -0,0 +1 @@ +Update vendorized expat library version to 2.2.6. diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h index 1f608c02d6fd..174c3fafda3f 100644 --- a/Modules/expat/expat.h +++ b/Modules/expat/expat.h @@ -264,7 +264,7 @@ XML_ParserCreate_MM(const XML_Char *encoding, const XML_Char *namespaceSeparator); /* Prepare a parser object to be re-used. This is particularly - valuable when memory allocation overhead is disproportionatly high, + valuable when memory allocation overhead is disproportionately high, such as when a large number of small documnents need to be parsed. All handlers are cleared from the parser, except for the unknownEncodingHandler. The parser's external state is re-initialized @@ -1076,7 +1076,7 @@ XML_GetFeatureList(void); */ #define XML_MAJOR_VERSION 2 #define XML_MINOR_VERSION 2 -#define XML_MICRO_VERSION 5 +#define XML_MICRO_VERSION 6 #ifdef __cplusplus } diff --git a/Modules/expat/siphash.h b/Modules/expat/siphash.h index 581872df7b49..4d6786d7839a 100644 --- a/Modules/expat/siphash.h +++ b/Modules/expat/siphash.h @@ -11,6 +11,12 @@ * -------------------------------------------------------------------------- * HISTORY: * + * 2018-07-08 (Anton Maklakov) + * - Add "fall through" markers for GCC's -Wimplicit-fallthrough + * + * 2017-11-03 (Sebastian Pipping) + * - Hide sip_tobin and sip_binof unless SIPHASH_TOBIN macro is defined + * * 2017-07-25 (Vadim Zeitlin) * - Fix use of SIPHASH_MAIN macro * @@ -151,6 +157,8 @@ static struct sipkey *sip_tokey(struct sipkey *key, const void *src) { } /* sip_tokey() */ +#ifdef SIPHASH_TOBIN + #define sip_binof(v) sip_tobin((unsigned char[8]){ 0 }, (v)) static void *sip_tobin(void *dst, uint64_t u64) { @@ -158,6 +166,8 @@ static void *sip_tobin(void *dst, uint64_t u64) { return dst; } /* sip_tobin() */ +#endif /* SIPHASH_TOBIN */ + static void sip_round(struct siphash *H, const int rounds) { int i; @@ -231,12 +241,19 @@ static uint64_t sip24_final(struct siphash *H) { switch (left) { case 7: b |= (uint64_t)H->buf[6] << 48; + /* fall through */ case 6: b |= (uint64_t)H->buf[5] << 40; + /* fall through */ case 5: b |= (uint64_t)H->buf[4] << 32; + /* fall through */ case 4: b |= (uint64_t)H->buf[3] << 24; + /* fall through */ case 3: b |= (uint64_t)H->buf[2] << 16; + /* fall through */ case 2: b |= (uint64_t)H->buf[1] << 8; + /* fall through */ case 1: b |= (uint64_t)H->buf[0] << 0; + /* fall through */ case 0: break; } diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 90a237f30eb8..c4f3ffc215c9 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -1,4 +1,4 @@ -/* 4b74aa710b4ed5ce464b0ce544852cb47bf905c85a49c7bae2749f5885cb966d (2.2.5+) +/* 19ac4776051591216f1874e34ee99b6a43a3784c8bd7d70efeb9258dd22b906a (2.2.6+) __ __ _ ___\ \/ /_ __ __ _| |_ / _ \\ /| '_ \ / _` | __| @@ -161,6 +161,9 @@ typedef char ICHAR; /* Round up n to be a multiple of sz, where sz is a power of 2. */ #define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1)) +/* Do safe (NULL-aware) pointer arithmetic */ +#define EXPAT_SAFE_PTR_DIFF(p, q) (((p) && (q)) ? ((p) - (q)) : 0) + /* Handle the case where memmove() doesn't exist. */ #ifndef HAVE_MEMMOVE #ifdef HAVE_BCOPY @@ -1820,6 +1823,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -1969,6 +1973,7 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -2026,39 +2031,46 @@ XML_GetBuffer(XML_Parser parser, int len) default: ; } - if (len > parser->m_bufferLim - parser->m_bufferEnd) { + if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) { #ifdef XML_CONTEXT_BYTES int keep; #endif /* defined XML_CONTEXT_BYTES */ /* Do not invoke signed arithmetic overflow: */ - int neededSize = (int) ((unsigned)len + (unsigned)(parser->m_bufferEnd - parser->m_bufferPtr)); + int neededSize = (int) ((unsigned)len + + (unsigned)EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, + parser->m_bufferPtr)); if (neededSize < 0) { parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } #ifdef XML_CONTEXT_BYTES - keep = (int)(parser->m_bufferPtr - parser->m_buffer); + keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; neededSize += keep; #endif /* defined XML_CONTEXT_BYTES */ - if (neededSize <= parser->m_bufferLim - parser->m_buffer) { + if (neededSize <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) { #ifdef XML_CONTEXT_BYTES - if (keep < parser->m_bufferPtr - parser->m_buffer) { - int offset = (int)(parser->m_bufferPtr - parser->m_buffer) - keep; + if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) { + int offset = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer) - keep; + /* The buffer pointers cannot be NULL here; we have at least some bytes in the buffer */ memmove(parser->m_buffer, &parser->m_buffer[offset], parser->m_bufferEnd - parser->m_bufferPtr + keep); parser->m_bufferEnd -= offset; parser->m_bufferPtr -= offset; } #else - memmove(parser->m_buffer, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferPtr = parser->m_buffer; + if (parser->m_buffer && parser->m_bufferPtr) { + memmove(parser->m_buffer, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + parser->m_bufferPtr = parser->m_buffer; + } #endif /* not defined XML_CONTEXT_BYTES */ } else { char *newBuf; - int bufferSize = (int)(parser->m_bufferLim - parser->m_bufferPtr); + int bufferSize = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr); if (bufferSize == 0) bufferSize = INIT_BUFFER_SIZE; do { @@ -2077,25 +2089,34 @@ XML_GetBuffer(XML_Parser parser, int len) parser->m_bufferLim = newBuf + bufferSize; #ifdef XML_CONTEXT_BYTES if (parser->m_bufferPtr) { - int keep = (int)(parser->m_bufferPtr - parser->m_buffer); + int keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; - memcpy(newBuf, &parser->m_bufferPtr[-keep], parser->m_bufferEnd - parser->m_bufferPtr + keep); + memcpy(newBuf, &parser->m_bufferPtr[-keep], + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep); FREE(parser, parser->m_buffer); parser->m_buffer = newBuf; - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr) + keep; + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep; parser->m_bufferPtr = parser->m_buffer + keep; } else { - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; parser->m_bufferPtr = parser->m_buffer = newBuf; } #else if (parser->m_bufferPtr) { - memcpy(newBuf, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); + memcpy(newBuf, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); FREE(parser, parser->m_buffer); + parser->m_bufferEnd = newBuf + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + } + else { + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; } - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); parser->m_bufferPtr = parser->m_buffer = newBuf; #endif /* not defined XML_CONTEXT_BYTES */ } @@ -2908,9 +2929,11 @@ doContent(XML_Parser parser, poolClear(&parser->m_tempPool); freeBindings(parser, bindings); } - if ((parser->m_tagLevel == 0) && - !((parser->m_parsingStatus.parsing == XML_FINISHED) || (parser->m_parsingStatus.parsing == XML_SUSPENDED))) { - return epilogProcessor(parser, next, end, nextPtr); + if ((parser->m_tagLevel == 0) && (parser->m_parsingStatus.parsing != XML_FINISHED)) { + if (parser->m_parsingStatus.parsing == XML_SUSPENDED) + parser->m_processor = epilogProcessor; + else + return epilogProcessor(parser, next, end, nextPtr); } break; case XML_TOK_END_TAG: @@ -4746,8 +4769,8 @@ doProlog(XML_Parser parser, return XML_ERROR_NO_MEMORY; parser->m_declEntity->publicId = NULL; } - /* fall through */ #endif /* XML_DTD */ + /* fall through */ case XML_ROLE_ENTITY_SYSTEM_ID: if (dtd->keepProcessing && parser->m_declEntity) { parser->m_declEntity->systemId = poolStoreString(&dtd->pool, enc, @@ -6643,7 +6666,6 @@ hash(XML_Parser parser, KEY s) { struct siphash state; struct sipkey key; - (void)sip_tobin; (void)sip24_valid; copy_salt_to_sipkey(parser, &key); sip24_init(&state, &key); diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c index 0403dd3d09a0..4d9ae7dc3896 100644 --- a/Modules/expat/xmltok_impl.c +++ b/Modules/expat/xmltok_impl.c @@ -74,6 +74,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ case BT_DIGIT: \ @@ -102,6 +103,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ ptr += MINBPC(enc); \ @@ -602,7 +604,7 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end, return XML_TOK_INVALID; } } - /* fall through */ + /* fall through */ case BT_EQUALS: { int open; @@ -1442,6 +1444,7 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end, case BT_NMSTRT: if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f)) break; + /* fall through */ default: switch (BYTE_TO_ASCII(enc, ptr)) { case 0x24: /* $ */ @@ -1659,8 +1662,8 @@ PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1, { for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) { if (end1 - ptr1 < MINBPC(enc)) { - /* This line cannot be executed. THe incoming data has already - * been tokenized once, so imcomplete characters like this have + /* This line cannot be executed. The incoming data has already + * been tokenized once, so incomplete characters like this have * already been eliminated from the input. Retaining the * paranoia check is still valuable, however. */ From webhook-mailer at python.org Tue Sep 11 00:56:53 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 04:56:53 -0000 Subject: [Python-checkins] bpo-34625: Update vendorized expat version to 2.2.6. (GH-9158) Message-ID: https://github.com/python/cpython/commit/b2260e59ff1eaf20de4738099005ddf507b7b27d commit: b2260e59ff1eaf20de4738099005ddf507b7b27d branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Benjamin Peterson date: 2018-09-10T21:56:50-07:00 summary: bpo-34625: Update vendorized expat version to 2.2.6. (GH-9158) (cherry picked from commit 5033aa77aacaa5505636f150e8d54baac5bdca9c) Co-authored-by: Benjamin Peterson files: A Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst M Modules/expat/expat.h M Modules/expat/siphash.h M Modules/expat/xmlparse.c M Modules/expat/xmltok_impl.c diff --git a/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst new file mode 100644 index 000000000000..0747ec54470f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-17-46-51.bpo-34625.D2YfDz.rst @@ -0,0 +1 @@ +Update vendorized expat library version to 2.2.6. diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h index 1f608c02d6fd..174c3fafda3f 100644 --- a/Modules/expat/expat.h +++ b/Modules/expat/expat.h @@ -264,7 +264,7 @@ XML_ParserCreate_MM(const XML_Char *encoding, const XML_Char *namespaceSeparator); /* Prepare a parser object to be re-used. This is particularly - valuable when memory allocation overhead is disproportionatly high, + valuable when memory allocation overhead is disproportionately high, such as when a large number of small documnents need to be parsed. All handlers are cleared from the parser, except for the unknownEncodingHandler. The parser's external state is re-initialized @@ -1076,7 +1076,7 @@ XML_GetFeatureList(void); */ #define XML_MAJOR_VERSION 2 #define XML_MINOR_VERSION 2 -#define XML_MICRO_VERSION 5 +#define XML_MICRO_VERSION 6 #ifdef __cplusplus } diff --git a/Modules/expat/siphash.h b/Modules/expat/siphash.h index 581872df7b49..4d6786d7839a 100644 --- a/Modules/expat/siphash.h +++ b/Modules/expat/siphash.h @@ -11,6 +11,12 @@ * -------------------------------------------------------------------------- * HISTORY: * + * 2018-07-08 (Anton Maklakov) + * - Add "fall through" markers for GCC's -Wimplicit-fallthrough + * + * 2017-11-03 (Sebastian Pipping) + * - Hide sip_tobin and sip_binof unless SIPHASH_TOBIN macro is defined + * * 2017-07-25 (Vadim Zeitlin) * - Fix use of SIPHASH_MAIN macro * @@ -151,6 +157,8 @@ static struct sipkey *sip_tokey(struct sipkey *key, const void *src) { } /* sip_tokey() */ +#ifdef SIPHASH_TOBIN + #define sip_binof(v) sip_tobin((unsigned char[8]){ 0 }, (v)) static void *sip_tobin(void *dst, uint64_t u64) { @@ -158,6 +166,8 @@ static void *sip_tobin(void *dst, uint64_t u64) { return dst; } /* sip_tobin() */ +#endif /* SIPHASH_TOBIN */ + static void sip_round(struct siphash *H, const int rounds) { int i; @@ -231,12 +241,19 @@ static uint64_t sip24_final(struct siphash *H) { switch (left) { case 7: b |= (uint64_t)H->buf[6] << 48; + /* fall through */ case 6: b |= (uint64_t)H->buf[5] << 40; + /* fall through */ case 5: b |= (uint64_t)H->buf[4] << 32; + /* fall through */ case 4: b |= (uint64_t)H->buf[3] << 24; + /* fall through */ case 3: b |= (uint64_t)H->buf[2] << 16; + /* fall through */ case 2: b |= (uint64_t)H->buf[1] << 8; + /* fall through */ case 1: b |= (uint64_t)H->buf[0] << 0; + /* fall through */ case 0: break; } diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 90a237f30eb8..c4f3ffc215c9 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -1,4 +1,4 @@ -/* 4b74aa710b4ed5ce464b0ce544852cb47bf905c85a49c7bae2749f5885cb966d (2.2.5+) +/* 19ac4776051591216f1874e34ee99b6a43a3784c8bd7d70efeb9258dd22b906a (2.2.6+) __ __ _ ___\ \/ /_ __ __ _| |_ / _ \\ /| '_ \ / _` | __| @@ -161,6 +161,9 @@ typedef char ICHAR; /* Round up n to be a multiple of sz, where sz is a power of 2. */ #define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1)) +/* Do safe (NULL-aware) pointer arithmetic */ +#define EXPAT_SAFE_PTR_DIFF(p, q) (((p) && (q)) ? ((p) - (q)) : 0) + /* Handle the case where memmove() doesn't exist. */ #ifndef HAVE_MEMMOVE #ifdef HAVE_BCOPY @@ -1820,6 +1823,7 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -1969,6 +1973,7 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal) parser->m_errorCode = XML_ERROR_NO_MEMORY; return XML_STATUS_ERROR; } + /* fall through */ default: parser->m_parsingStatus.parsing = XML_PARSING; } @@ -2026,39 +2031,46 @@ XML_GetBuffer(XML_Parser parser, int len) default: ; } - if (len > parser->m_bufferLim - parser->m_bufferEnd) { + if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) { #ifdef XML_CONTEXT_BYTES int keep; #endif /* defined XML_CONTEXT_BYTES */ /* Do not invoke signed arithmetic overflow: */ - int neededSize = (int) ((unsigned)len + (unsigned)(parser->m_bufferEnd - parser->m_bufferPtr)); + int neededSize = (int) ((unsigned)len + + (unsigned)EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, + parser->m_bufferPtr)); if (neededSize < 0) { parser->m_errorCode = XML_ERROR_NO_MEMORY; return NULL; } #ifdef XML_CONTEXT_BYTES - keep = (int)(parser->m_bufferPtr - parser->m_buffer); + keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; neededSize += keep; #endif /* defined XML_CONTEXT_BYTES */ - if (neededSize <= parser->m_bufferLim - parser->m_buffer) { + if (neededSize <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) { #ifdef XML_CONTEXT_BYTES - if (keep < parser->m_bufferPtr - parser->m_buffer) { - int offset = (int)(parser->m_bufferPtr - parser->m_buffer) - keep; + if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) { + int offset = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer) - keep; + /* The buffer pointers cannot be NULL here; we have at least some bytes in the buffer */ memmove(parser->m_buffer, &parser->m_buffer[offset], parser->m_bufferEnd - parser->m_bufferPtr + keep); parser->m_bufferEnd -= offset; parser->m_bufferPtr -= offset; } #else - memmove(parser->m_buffer, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr); - parser->m_bufferPtr = parser->m_buffer; + if (parser->m_buffer && parser->m_bufferPtr) { + memmove(parser->m_buffer, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + parser->m_bufferPtr = parser->m_buffer; + } #endif /* not defined XML_CONTEXT_BYTES */ } else { char *newBuf; - int bufferSize = (int)(parser->m_bufferLim - parser->m_bufferPtr); + int bufferSize = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr); if (bufferSize == 0) bufferSize = INIT_BUFFER_SIZE; do { @@ -2077,25 +2089,34 @@ XML_GetBuffer(XML_Parser parser, int len) parser->m_bufferLim = newBuf + bufferSize; #ifdef XML_CONTEXT_BYTES if (parser->m_bufferPtr) { - int keep = (int)(parser->m_bufferPtr - parser->m_buffer); + int keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer); if (keep > XML_CONTEXT_BYTES) keep = XML_CONTEXT_BYTES; - memcpy(newBuf, &parser->m_bufferPtr[-keep], parser->m_bufferEnd - parser->m_bufferPtr + keep); + memcpy(newBuf, &parser->m_bufferPtr[-keep], + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep); FREE(parser, parser->m_buffer); parser->m_buffer = newBuf; - parser->m_bufferEnd = parser->m_buffer + (parser->m_bufferEnd - parser->m_bufferPtr) + keep; + parser->m_bufferEnd = parser->m_buffer + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr) + keep; parser->m_bufferPtr = parser->m_buffer + keep; } else { - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; parser->m_bufferPtr = parser->m_buffer = newBuf; } #else if (parser->m_bufferPtr) { - memcpy(newBuf, parser->m_bufferPtr, parser->m_bufferEnd - parser->m_bufferPtr); + memcpy(newBuf, parser->m_bufferPtr, + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)); FREE(parser, parser->m_buffer); + parser->m_bufferEnd = newBuf + + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr); + } + else { + /* This must be a brand new buffer with no data in it yet */ + parser->m_bufferEnd = newBuf; } - parser->m_bufferEnd = newBuf + (parser->m_bufferEnd - parser->m_bufferPtr); parser->m_bufferPtr = parser->m_buffer = newBuf; #endif /* not defined XML_CONTEXT_BYTES */ } @@ -2908,9 +2929,11 @@ doContent(XML_Parser parser, poolClear(&parser->m_tempPool); freeBindings(parser, bindings); } - if ((parser->m_tagLevel == 0) && - !((parser->m_parsingStatus.parsing == XML_FINISHED) || (parser->m_parsingStatus.parsing == XML_SUSPENDED))) { - return epilogProcessor(parser, next, end, nextPtr); + if ((parser->m_tagLevel == 0) && (parser->m_parsingStatus.parsing != XML_FINISHED)) { + if (parser->m_parsingStatus.parsing == XML_SUSPENDED) + parser->m_processor = epilogProcessor; + else + return epilogProcessor(parser, next, end, nextPtr); } break; case XML_TOK_END_TAG: @@ -4746,8 +4769,8 @@ doProlog(XML_Parser parser, return XML_ERROR_NO_MEMORY; parser->m_declEntity->publicId = NULL; } - /* fall through */ #endif /* XML_DTD */ + /* fall through */ case XML_ROLE_ENTITY_SYSTEM_ID: if (dtd->keepProcessing && parser->m_declEntity) { parser->m_declEntity->systemId = poolStoreString(&dtd->pool, enc, @@ -6643,7 +6666,6 @@ hash(XML_Parser parser, KEY s) { struct siphash state; struct sipkey key; - (void)sip_tobin; (void)sip24_valid; copy_salt_to_sipkey(parser, &key); sip24_init(&state, &key); diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c index 0403dd3d09a0..4d9ae7dc3896 100644 --- a/Modules/expat/xmltok_impl.c +++ b/Modules/expat/xmltok_impl.c @@ -74,6 +74,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ case BT_DIGIT: \ @@ -102,6 +103,7 @@ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ + /* fall through */ \ case BT_NMSTRT: \ case BT_HEX: \ ptr += MINBPC(enc); \ @@ -602,7 +604,7 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end, return XML_TOK_INVALID; } } - /* fall through */ + /* fall through */ case BT_EQUALS: { int open; @@ -1442,6 +1444,7 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end, case BT_NMSTRT: if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f)) break; + /* fall through */ default: switch (BYTE_TO_ASCII(enc, ptr)) { case 0x24: /* $ */ @@ -1659,8 +1662,8 @@ PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1, { for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) { if (end1 - ptr1 < MINBPC(enc)) { - /* This line cannot be executed. THe incoming data has already - * been tokenized once, so imcomplete characters like this have + /* This line cannot be executed. The incoming data has already + * been tokenized once, so incomplete characters like this have * already been eliminated from the input. Retaining the * paranoia check is still valuable, however. */ From webhook-mailer at python.org Tue Sep 11 01:12:45 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 05:12:45 -0000 Subject: [Python-checkins] closes bpo-33883: Mention type checkers in the FAQ. (GH-7760) Message-ID: https://github.com/python/cpython/commit/a37825418649873a0fa971dc7e5e6d142c124574 commit: a37825418649873a0fa971dc7e5e6d142c124574 branch: master author: Andr?s Delfino committer: Benjamin Peterson date: 2018-09-10T22:12:41-07:00 summary: closes bpo-33883: Mention type checkers in the FAQ. (GH-7760) files: M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 53f3b7f528c0..fd720c1a304b 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -71,6 +71,11 @@ length, whether variable names are well-formed according to your coding standard, whether declared interfaces are fully implemented, and more. https://docs.pylint.org/ provides a full list of Pylint's features. +Static type checkers such as `Mypy `_, +`Pyre `_, and +`Pytype `_ can check type hints in Python +source code. + How can I create a stand-alone binary from a Python script? ----------------------------------------------------------- From webhook-mailer at python.org Tue Sep 11 01:27:04 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 05:27:04 -0000 Subject: [Python-checkins] closes bpo-33883: Mention type checkers in the FAQ. (GH-7760) Message-ID: https://github.com/python/cpython/commit/c9ece5f4a961d4963ed42aefa934eb20f9a7222b commit: c9ece5f4a961d4963ed42aefa934eb20f9a7222b branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-10T22:27:00-07:00 summary: closes bpo-33883: Mention type checkers in the FAQ. (GH-7760) (cherry picked from commit a37825418649873a0fa971dc7e5e6d142c124574) Co-authored-by: Andr?s Delfino files: M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 53f3b7f528c0..fd720c1a304b 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -71,6 +71,11 @@ length, whether variable names are well-formed according to your coding standard, whether declared interfaces are fully implemented, and more. https://docs.pylint.org/ provides a full list of Pylint's features. +Static type checkers such as `Mypy `_, +`Pyre `_, and +`Pytype `_ can check type hints in Python +source code. + How can I create a stand-alone binary from a Python script? ----------------------------------------------------------- From webhook-mailer at python.org Tue Sep 11 02:00:54 2018 From: webhook-mailer at python.org (Gregory P. Smith) Date: Tue, 11 Sep 2018 06:00:54 -0000 Subject: [Python-checkins] bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242) (GH-9149) Message-ID: https://github.com/python/cpython/commit/2173bb818c6c726d831b106ed0d3fad7825905dc commit: 2173bb818c6c726d831b106ed0d3fad7825905dc branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Gregory P. Smith date: 2018-09-10T23:00:47-07:00 summary: bpo-32270: Don't close stdin/out/err in pass_fds (GH-6242) (GH-9149) When subprocess.Popen() stdin= stdout= or stderr= handles are specified and appear in pass_fds=, don't close the original fds after dup'ing them. This implementation and unittest primarily came from @izbyshev (see the PR) See also https://github.com/izbyshev/cpython/commit/b89b52f28490b69142d5c061604b3a3989cec66c This also removes the old manual p2cread, c2pwrite, and errwrite closing logic as inheritable flags and _close_open_fds takes care of that properly today without special treatment. This code is within child_exec() where it is the only thread so there is no race condition between the dup and _Py_set_inheritable_async_safe call. (cherry picked from commit ce34410b8b67f49d8275c05d51b3ead50cf97f48) Co-authored-by: Gregory P. Smith [Google] files: A Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst M Lib/test/test_subprocess.py M Modules/_posixsubprocess.c diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 9ebe3228ac03..e1b24778430e 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -2493,6 +2493,36 @@ def test_pass_fds_inheritable(self): self.assertEqual(os.get_inheritable(inheritable), True) self.assertEqual(os.get_inheritable(non_inheritable), False) + + # bpo-32270: Ensure that descriptors specified in pass_fds + # are inherited even if they are used in redirections. + # Contributed by @izbyshev. + def test_pass_fds_redirected(self): + """Regression test for https://bugs.python.org/issue32270.""" + fd_status = support.findfile("fd_status.py", subdir="subprocessdata") + pass_fds = [] + for _ in range(2): + fd = os.open(os.devnull, os.O_RDWR) + self.addCleanup(os.close, fd) + pass_fds.append(fd) + + stdout_r, stdout_w = os.pipe() + self.addCleanup(os.close, stdout_r) + self.addCleanup(os.close, stdout_w) + pass_fds.insert(1, stdout_w) + + with subprocess.Popen([sys.executable, fd_status], + stdin=pass_fds[0], + stdout=pass_fds[1], + stderr=pass_fds[2], + close_fds=True, + pass_fds=pass_fds): + output = os.read(stdout_r, 1024) + fds = {int(num) for num in output.split(b',')} + + self.assertEqual(fds, {0, 1, 2} | frozenset(pass_fds), f"output={output!a}") + + def test_stdout_stdin_are_single_inout_fd(self): with io.open(os.devnull, "r+") as inout: p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"], diff --git a/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst b/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst new file mode 100644 index 000000000000..83f68624c1be --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-14-15-53.bpo-32270.wSJjuD.rst @@ -0,0 +1,2 @@ +The subprocess module no longer mistakenly closes redirected fds even when +they were in pass_fds when outside of the default {0, 1, 2} set. diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index ad934dfe7d87..fe0e554618aa 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -422,10 +422,20 @@ child_exec(char *const exec_array[], /* When duping fds, if there arises a situation where one of the fds is either 0, 1 or 2, it is possible that it is overwritten (#12607). */ - if (c2pwrite == 0) + if (c2pwrite == 0) { POSIX_CALL(c2pwrite = dup(c2pwrite)); - while (errwrite == 0 || errwrite == 1) + /* issue32270 */ + if (_Py_set_inheritable_async_safe(c2pwrite, 0, NULL) < 0) { + goto error; + } + } + while (errwrite == 0 || errwrite == 1) { POSIX_CALL(errwrite = dup(errwrite)); + /* issue32270 */ + if (_Py_set_inheritable_async_safe(errwrite, 0, NULL) < 0) { + goto error; + } + } /* Dup fds for child. dup2() removes the CLOEXEC flag but we must do it ourselves if dup2() @@ -451,14 +461,8 @@ child_exec(char *const exec_array[], else if (errwrite != -1) POSIX_CALL(dup2(errwrite, 2)); /* stderr */ - /* Close pipe fds. Make sure we don't close the same fd more than */ - /* once, or standard fds. */ - if (p2cread > 2) - POSIX_CALL(close(p2cread)); - if (c2pwrite > 2 && c2pwrite != p2cread) - POSIX_CALL(close(c2pwrite)); - if (errwrite != c2pwrite && errwrite != p2cread && errwrite > 2) - POSIX_CALL(close(errwrite)); + /* We no longer manually close p2cread, c2pwrite, and errwrite here as + * _close_open_fds takes care when it is not already non-inheritable. */ if (cwd) POSIX_CALL(chdir(cwd)); From solipsis at pitrou.net Tue Sep 11 05:10:44 2018 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 11 Sep 2018 09:10:44 +0000 Subject: [Python-checkins] Daily reference leaks (4243df51fe43): sum=3 Message-ID: <20180911091044.1.C1EB1FBAC2FAE48C@psf.io> results for 4243df51fe43 on branch "default" -------------------------------------------- test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_forkserver leaked [0, -2, 1] memory blocks, sum=-1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogpFMo57', '--timeout', '7200'] From webhook-mailer at python.org Tue Sep 11 11:52:43 2018 From: webhook-mailer at python.org (Ned Deily) Date: Tue, 11 Sep 2018 15:52:43 -0000 Subject: [Python-checkins] bpo-34405: Update to OpenSSL 1.1.0i for macOS installer builds (GH-9166) Message-ID: https://github.com/python/cpython/commit/3102e24d83315eee42a94c460956fbcb92ac510f commit: 3102e24d83315eee42a94c460956fbcb92ac510f branch: master author: Ned Deily committer: GitHub date: 2018-09-11T08:52:40-07:00 summary: bpo-34405: Update to OpenSSL 1.1.0i for macOS installer builds (GH-9166) files: A Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index d2b04d163ab4..15488780327e 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -215,9 +215,9 @@ def library_recipes(): result.extend([ dict( - name="OpenSSL 1.1.0h", - url="https://www.openssl.org/source/openssl-1.1.0h.tar.gz", - checksum='5271477e4d93f4ea032b665ef095ff24', + name="OpenSSL 1.1.0i", + url="https://www.openssl.org/source/openssl-1.1.0i.tar.gz", + checksum='9495126aafd2659d357ea66a969c3fe1', buildrecipe=build_universal_openssl, configure=None, install=None, diff --git a/Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst b/Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst new file mode 100644 index 000000000000..3bc9c4c2a037 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst @@ -0,0 +1 @@ +Update to OpenSSL 1.1.0i for macOS installer builds. From webhook-mailer at python.org Tue Sep 11 11:54:10 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 15:54:10 -0000 Subject: [Python-checkins] closes bpo-32490: Fix filename duplication in subprocess exception message. (GH-9163) Message-ID: https://github.com/python/cpython/commit/73870bfeb9cf350d84ee88bd25430c104b3c6191 commit: 73870bfeb9cf350d84ee88bd25430c104b3c6191 branch: master author: Zackery Spytz committer: Benjamin Peterson date: 2018-09-11T08:54:07-07:00 summary: closes bpo-32490: Fix filename duplication in subprocess exception message. (GH-9163) 8621bb5d93239316f97281826461b85072ff6db7 sets the filename in directly in the FileNotFoundError, so we may revert the earlier fix 5f780400572508a8179de6a6c13b58b7be417ef5. files: A Misc/NEWS.d/next/Library/2018-09-11-01-25-35.bpo-32490.ROIDO1.rst M Lib/subprocess.py M Lib/test/test_subprocess.py diff --git a/Lib/subprocess.py b/Lib/subprocess.py index 1e04d5e57013..c827113a933f 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1512,8 +1512,6 @@ def _execute_child(self, args, executable, preexec_fn, close_fds, err_filename = orig_executable if errno_num != 0: err_msg = os.strerror(errno_num) - if errno_num == errno.ENOENT: - err_msg += ': ' + repr(err_filename) raise child_exception_type(errno_num, err_msg, err_filename) raise child_exception_type(err_msg) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 8419061b2a90..c56e1b632150 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1520,7 +1520,6 @@ def _get_chdir_exception(self): # string and instead capture the exception that we want to see # below for comparison. desired_exception = e - desired_exception.strerror += ': ' + repr(self._nonexistent_dir) else: self.fail("chdir to nonexistent directory %s succeeded." % self._nonexistent_dir) @@ -1537,6 +1536,7 @@ def test_exception_cwd(self): # it up to the parent process as the correct exception. self.assertEqual(desired_exception.errno, e.errno) self.assertEqual(desired_exception.strerror, e.strerror) + self.assertEqual(desired_exception.filename, e.filename) else: self.fail("Expected OSError: %s" % desired_exception) @@ -1551,6 +1551,7 @@ def test_exception_bad_executable(self): # it up to the parent process as the correct exception. self.assertEqual(desired_exception.errno, e.errno) self.assertEqual(desired_exception.strerror, e.strerror) + self.assertEqual(desired_exception.filename, e.filename) else: self.fail("Expected OSError: %s" % desired_exception) @@ -1564,6 +1565,7 @@ def test_exception_bad_args_0(self): # it up to the parent process as the correct exception. self.assertEqual(desired_exception.errno, e.errno) self.assertEqual(desired_exception.strerror, e.strerror) + self.assertEqual(desired_exception.filename, e.filename) else: self.fail("Expected OSError: %s" % desired_exception) diff --git a/Misc/NEWS.d/next/Library/2018-09-11-01-25-35.bpo-32490.ROIDO1.rst b/Misc/NEWS.d/next/Library/2018-09-11-01-25-35.bpo-32490.ROIDO1.rst new file mode 100644 index 000000000000..16fe7b4d4c09 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-11-01-25-35.bpo-32490.ROIDO1.rst @@ -0,0 +1,2 @@ +Prevent filename duplication in :mod:`subprocess` exception messages. Patch +by Zackery Spytz. From webhook-mailer at python.org Tue Sep 11 12:07:16 2018 From: webhook-mailer at python.org (Ned Deily) Date: Tue, 11 Sep 2018 16:07:16 -0000 Subject: [Python-checkins] bpo-34405: Update to OpenSSL 1.1.0i for macOS installer builds (GH-9166) (GH-9167) Message-ID: https://github.com/python/cpython/commit/3235fac0d7d94ad6464a162261c18a424829d2e5 commit: 3235fac0d7d94ad6464a162261c18a424829d2e5 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Ned Deily date: 2018-09-11T09:07:11-07:00 summary: bpo-34405: Update to OpenSSL 1.1.0i for macOS installer builds (GH-9166) (GH-9167) (cherry picked from commit 3102e24d83315eee42a94c460956fbcb92ac510f) Co-authored-by: Ned Deily files: A Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index d2b04d163ab4..15488780327e 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -215,9 +215,9 @@ def library_recipes(): result.extend([ dict( - name="OpenSSL 1.1.0h", - url="https://www.openssl.org/source/openssl-1.1.0h.tar.gz", - checksum='5271477e4d93f4ea032b665ef095ff24', + name="OpenSSL 1.1.0i", + url="https://www.openssl.org/source/openssl-1.1.0i.tar.gz", + checksum='9495126aafd2659d357ea66a969c3fe1', buildrecipe=build_universal_openssl, configure=None, install=None, diff --git a/Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst b/Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst new file mode 100644 index 000000000000..3bc9c4c2a037 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2018-09-11-08-30-55.bpo-34405.UzIi0n.rst @@ -0,0 +1 @@ +Update to OpenSSL 1.1.0i for macOS installer builds. From webhook-mailer at python.org Tue Sep 11 12:51:32 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 16:51:32 -0000 Subject: [Python-checkins] closes bpo-29832: Remove "getsockaddrarg" from error messages. (GH-3163) Message-ID: https://github.com/python/cpython/commit/735171e33486131d93865cf851c0c3d63fffd364 commit: 735171e33486131d93865cf851c0c3d63fffd364 branch: master author: Oren Milman committer: Benjamin Peterson date: 2018-09-11T09:51:29-07:00 summary: closes bpo-29832: Remove "getsockaddrarg" from error messages. (GH-3163) files: A Misc/NEWS.d/next/Core and Builtins/2017-09-12-08-11-01.bpo-29832.Kuf2M7.rst M Modules/socketmodule.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-09-12-08-11-01.bpo-29832.Kuf2M7.rst b/Misc/NEWS.d/next/Core and Builtins/2017-09-12-08-11-01.bpo-29832.Kuf2M7.rst new file mode 100644 index 000000000000..6cf6696a71f7 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2017-09-12-08-11-01.bpo-29832.Kuf2M7.rst @@ -0,0 +1,2 @@ +Remove references to 'getsockaddrarg' from various socket error messages. +Patch by Oren Milman. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 014f3ba8b5ea..8aadc780ffeb 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1583,7 +1583,7 @@ idna_converter(PyObject *obj, struct maybe_idna *data) static int getsockaddrarg(PySocketSockObject *s, PyObject *args, - struct sockaddr *addr_ret, int *len_ret) + struct sockaddr *addr_ret, int *len_ret, const char *caller) { switch (s->sock_family) { @@ -1649,13 +1649,17 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (!PyTuple_Check(args)) { PyErr_Format( PyExc_TypeError, - "getsockaddrarg: " - "AF_NETLINK address must be tuple, not %.500s", - Py_TYPE(args)->tp_name); + "%s(): AF_NETLINK address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); return 0; } - if (!PyArg_ParseTuple(args, "II:getsockaddrarg", &pid, &groups)) + if (!PyArg_ParseTuple(args, + "II;AF_NETLINK address must be a pair " + "(pid, groups)", + &pid, &groups)) + { return 0; + } addr->nl_family = AF_NETLINK; addr->nl_pid = pid; addr->nl_groups = groups; @@ -1703,14 +1707,22 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (!PyTuple_Check(args)) { PyErr_Format( PyExc_TypeError, - "getsockaddrarg: " - "AF_INET address must be tuple, not %.500s", - Py_TYPE(args)->tp_name); + "%s(): AF_INET address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); return 0; } - if (!PyArg_ParseTuple(args, "O&i:getsockaddrarg", + if (!PyArg_ParseTuple(args, + "O&i;AF_INET address must be a pair " + "(host, port)", idna_converter, &host, &port)) + { + assert(PyErr_Occurred()); + if (PyErr_ExceptionMatches(PyExc_OverflowError)) { + PyErr_Format(PyExc_OverflowError, + "%s(): port must be 0-65535.", caller); + } return 0; + } addr=(struct sockaddr_in*)addr_ret; result = setipaddr(host.buf, (struct sockaddr *)addr, sizeof(*addr), AF_INET); @@ -1718,9 +1730,9 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (result < 0) return 0; if (port < 0 || port > 0xffff) { - PyErr_SetString( + PyErr_Format( PyExc_OverflowError, - "getsockaddrarg: port must be 0-65535."); + "%s(): port must be 0-65535.", caller); return 0; } addr->sin_family = AF_INET; @@ -1740,14 +1752,21 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (!PyTuple_Check(args)) { PyErr_Format( PyExc_TypeError, - "getsockaddrarg: " - "AF_INET6 address must be tuple, not %.500s", - Py_TYPE(args)->tp_name); + "%s(): AF_INET6 address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); return 0; } - if (!PyArg_ParseTuple(args, "O&i|II", + if (!PyArg_ParseTuple(args, + "O&i|II;AF_INET6 address must be a tuple " + "(host, port[, flowinfo[, scopeid]])", idna_converter, &host, &port, &flowinfo, - &scope_id)) { + &scope_id)) + { + assert(PyErr_Occurred()); + if (PyErr_ExceptionMatches(PyExc_OverflowError)) { + PyErr_Format(PyExc_OverflowError, + "%s(): port must be 0-65535.", caller); + } return 0; } addr = (struct sockaddr_in6*)addr_ret; @@ -1757,15 +1776,15 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (result < 0) return 0; if (port < 0 || port > 0xffff) { - PyErr_SetString( + PyErr_Format( PyExc_OverflowError, - "getsockaddrarg: port must be 0-65535."); + "%s(): port must be 0-65535.", caller); return 0; } if (flowinfo > 0xfffff) { - PyErr_SetString( + PyErr_Format( PyExc_OverflowError, - "getsockaddrarg: flowinfo must be 0-1048575."); + "%s(): flowinfo must be 0-1048575.", caller); return 0; } addr->sin6_family = s->sock_family; @@ -1791,8 +1810,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, _BT_L2_MEMB(addr, family) = AF_BLUETOOTH; if (!PyArg_ParseTuple(args, "si", &straddr, &_BT_L2_MEMB(addr, psm))) { - PyErr_SetString(PyExc_OSError, "getsockaddrarg: " - "wrong format"); + PyErr_Format(PyExc_OSError, + "%s(): wrong format", caller); return 0; } if (setbdaddr(straddr, &_BT_L2_MEMB(addr, bdaddr)) < 0) @@ -1810,8 +1829,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, _BT_RC_MEMB(addr, family) = AF_BLUETOOTH; if (!PyArg_ParseTuple(args, "si", &straddr, &_BT_RC_MEMB(addr, channel))) { - PyErr_SetString(PyExc_OSError, "getsockaddrarg: " - "wrong format"); + PyErr_Format(PyExc_OSError, + "%s(): wrong format", caller); return 0; } if (setbdaddr(straddr, &_BT_RC_MEMB(addr, bdaddr)) < 0) @@ -1827,8 +1846,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, const char *straddr; _BT_HCI_MEMB(addr, family) = AF_BLUETOOTH; if (!PyBytes_Check(args)) { - PyErr_SetString(PyExc_OSError, "getsockaddrarg: " - "wrong format"); + PyErr_Format(PyExc_OSError, "%s: " + "wrong format", caller); return 0; } straddr = PyBytes_AS_STRING(args); @@ -1837,8 +1856,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, #else /* __NetBSD__ || __DragonFly__ */ _BT_HCI_MEMB(addr, family) = AF_BLUETOOTH; if (!PyArg_ParseTuple(args, "i", &_BT_HCI_MEMB(addr, dev))) { - PyErr_SetString(PyExc_OSError, "getsockaddrarg: " - "wrong format"); + PyErr_Format(PyExc_OSError, + "%s(): wrong format", caller); return 0; } #endif /* !(__NetBSD__ || __DragonFly__) */ @@ -1854,8 +1873,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, addr = (struct sockaddr_sco *)addr_ret; _BT_SCO_MEMB(addr, family) = AF_BLUETOOTH; if (!PyBytes_Check(args)) { - PyErr_SetString(PyExc_OSError, "getsockaddrarg: " - "wrong format"); + PyErr_Format(PyExc_OSError, + "%s(): wrong format", caller); return 0; } straddr = PyBytes_AS_STRING(args); @@ -1867,7 +1886,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, } #endif /* !__FreeBSD__ */ default: - PyErr_SetString(PyExc_OSError, "getsockaddrarg: unknown Bluetooth protocol"); + PyErr_Format(PyExc_OSError, + "%s(): unknown Bluetooth protocol", caller); return 0; } } @@ -1887,15 +1907,26 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (!PyTuple_Check(args)) { PyErr_Format( PyExc_TypeError, - "getsockaddrarg: " - "AF_PACKET address must be tuple, not %.500s", - Py_TYPE(args)->tp_name); + "%s(): AF_PACKET address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); return 0; } - if (!PyArg_ParseTuple(args, "si|iiy*", &interfaceName, - &protoNumber, &pkttype, &hatype, + /* XXX: improve the default error message according to the + documentation of AF_PACKET, which would be added as part + of bpo-25041. */ + if (!PyArg_ParseTuple(args, + "si|iiy*;AF_PACKET address must be a tuple of " + "two to five elements", + &interfaceName, &protoNumber, &pkttype, &hatype, &haddr)) + { + assert(PyErr_Occurred()); + if (PyErr_ExceptionMatches(PyExc_OverflowError)) { + PyErr_Format(PyExc_OverflowError, + "%s(): address argument out of range", caller); + } return 0; + } strncpy(ifr.ifr_name, interfaceName, sizeof(ifr.ifr_name)); ifr.ifr_name[(sizeof(ifr.ifr_name))-1] = '\0'; if (ioctl(s->sock_fd, SIOCGIFINDEX, &ifr) < 0) { @@ -1910,9 +1941,9 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, return 0; } if (protoNumber < 0 || protoNumber > 0xffff) { - PyErr_SetString( + PyErr_Format( PyExc_OverflowError, - "getsockaddrarg: protoNumber must be 0-65535."); + "%s(): protoNumber must be 0-65535.", caller); PyBuffer_Release(&haddr); return 0; } @@ -1944,16 +1975,18 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, if (!PyTuple_Check(args)) { PyErr_Format( PyExc_TypeError, - "getsockaddrarg: " - "AF_TIPC address must be tuple, not %.500s", - Py_TYPE(args)->tp_name); + "%s(): AF_TIPC address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); return 0; } if (!PyArg_ParseTuple(args, - "IIII|I;Invalid TIPC address format", - &atype, &v1, &v2, &v3, &scope)) + "IIII|I;AF_TIPC address must be a tuple " + "(addr_type, v1, v2, v3[, scope])", + &atype, &v1, &v2, &v3, &scope)) + { return 0; + } addr = (struct sockaddr_tipc *) addr_ret; memset(addr, 0, sizeof(struct sockaddr_tipc)); @@ -2002,9 +2035,19 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, Py_ssize_t len; addr = (struct sockaddr_can *)addr_ret; - if (!PyArg_ParseTuple(args, "O&", PyUnicode_FSConverter, - &interfaceName)) + if (!PyTuple_Check(args)) { + PyErr_Format(PyExc_TypeError, + "%s(): AF_CAN address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); + return 0; + } + if (!PyArg_ParseTuple(args, + "O&;AF_CAN address must be a tuple " + "(interface, )", + PyUnicode_FSConverter, &interfaceName)) + { return 0; + } len = PyBytes_GET_SIZE(interfaceName); @@ -2081,8 +2124,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, } #endif /* CAN_ISOTP */ default: - PyErr_SetString(PyExc_OSError, - "getsockaddrarg: unsupported CAN protocol"); + PyErr_Format(PyExc_OSError, + "%s(): unsupported CAN protocol", caller); return 0; } #endif /* AF_CAN && SIOCGIFINDEX */ @@ -2128,9 +2171,9 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, addr->sc_unit = 0; } else if (!PyArg_ParseTuple(args, "II", &(addr->sc_id), &(addr->sc_unit))) { - PyErr_SetString(PyExc_TypeError, "getsockaddrarg: " - "expected str or tuple of two ints"); - + PyErr_Format(PyExc_TypeError, + "%s(): PF_SYSTEM address must be a str or " + "a pair (id, unit)", caller); return 0; } @@ -2139,8 +2182,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, } #endif /* SYSPROTO_CONTROL */ default: - PyErr_SetString(PyExc_OSError, - "getsockaddrarg: unsupported PF_SYSTEM protocol"); + PyErr_Format(PyExc_OSError, + "%s(): unsupported PF_SYSTEM protocol", caller); return 0; } #endif /* PF_SYSTEM */ @@ -2155,9 +2198,19 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, memset(sa, 0, sizeof(*sa)); sa->salg_family = AF_ALG; - if (!PyArg_ParseTuple(args, "ss|HH:getsockaddrarg", - &type, &name, &sa->salg_feat, &sa->salg_mask)) + if (!PyTuple_Check(args)) { + PyErr_Format(PyExc_TypeError, + "%s(): AF_ALG address must be tuple, not %.500s", + caller, Py_TYPE(args)->tp_name); + return 0; + } + if (!PyArg_ParseTuple(args, + "ss|HH;AF_ALG address must be a tuple " + "(type, name[, feat[, mask]])", + &type, &name, &sa->salg_feat, &sa->salg_mask)) + { return 0; + } /* sockaddr_alg has fixed-sized char arrays for type and name */ if (strlen(type) > sizeof(sa->salg_type)) { PyErr_SetString(PyExc_ValueError, "AF_ALG type too long."); @@ -2178,7 +2231,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, /* More cases here... */ default: - PyErr_SetString(PyExc_OSError, "getsockaddrarg: bad family"); + PyErr_Format(PyExc_OSError, "%s(): bad family", caller); return 0; } @@ -2910,8 +2963,9 @@ sock_bind(PySocketSockObject *s, PyObject *addro) int addrlen; int res; - if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen)) + if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "bind")) { return NULL; + } Py_BEGIN_ALLOW_THREADS res = bind(s->sock_fd, SAS2SA(&addrbuf), addrlen); Py_END_ALLOW_THREADS @@ -3074,8 +3128,9 @@ sock_connect(PySocketSockObject *s, PyObject *addro) int addrlen; int res; - if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen)) + if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "connect")) { return NULL; + } res = internal_connect(s, SAS2SA(&addrbuf), addrlen, 1); if (res < 0) @@ -3100,8 +3155,9 @@ sock_connect_ex(PySocketSockObject *s, PyObject *addro) int addrlen; int res; - if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen)) + if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "connect_ex")) { return NULL; + } res = internal_connect(s, SAS2SA(&addrbuf), addrlen, 0); if (res < 0) @@ -4100,7 +4156,7 @@ sock_sendto(PySocketSockObject *s, PyObject *args) return select_error(); } - if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen)) { + if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "sendto")) { PyBuffer_Release(&pbuf); return NULL; } @@ -4231,8 +4287,11 @@ sock_sendmsg(PySocketSockObject *s, PyObject *args) /* Parse destination address. */ if (addr_arg != NULL && addr_arg != Py_None) { - if (!getsockaddrarg(s, addr_arg, SAS2SA(&addrbuf), &addrlen)) + if (!getsockaddrarg(s, addr_arg, SAS2SA(&addrbuf), &addrlen, + "sendmsg")) + { goto finally; + } msg.msg_name = &addrbuf; msg.msg_namelen = addrlen; } From webhook-mailer at python.org Tue Sep 11 12:54:45 2018 From: webhook-mailer at python.org (Yury Selivanov) Date: Tue, 11 Sep 2018 16:54:45 -0000 Subject: [Python-checkins] bpo-33649: First asyncio docs improvement pass (GH-9142) Message-ID: https://github.com/python/cpython/commit/7c7605ff1133cf757cac428c483827f666c7c827 commit: 7c7605ff1133cf757cac428c483827f666c7c827 branch: master author: Yury Selivanov committer: GitHub date: 2018-09-11T09:54:40-07:00 summary: bpo-33649: First asyncio docs improvement pass (GH-9142) Rewritten/updated sections: * Event Loop APIs * Transports & Protocols * Streams * Exceptions * Policies * Queues * Subprocesses * Platforms files: A Doc/library/asyncio-exceptions.rst A Doc/library/asyncio-platforms.rst A Doc/library/asyncio-policy.rst D Doc/library/asyncio-eventloops.rst M Doc/library/asyncio-dev.rst M Doc/library/asyncio-eventloop.rst M Doc/library/asyncio-protocol.rst M Doc/library/asyncio-queue.rst M Doc/library/asyncio-stream.rst M Doc/library/asyncio-subprocess.rst M Doc/library/asyncio-task.rst M Doc/library/asyncio.rst M Doc/library/ipc.rst M Doc/whatsnew/3.5.rst M Doc/whatsnew/3.6.rst M Doc/whatsnew/3.7.rst M Doc/whatsnew/3.8.rst diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 100fff561c5b..cb574c308ad1 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -23,7 +23,7 @@ To enable all debug checks for an application: * Enable the asyncio debug mode globally by setting the environment variable :envvar:`PYTHONASYNCIODEBUG` to ``1``, using ``-X dev`` command line option (see the :option:`-X` option), or by calling - :meth:`AbstractEventLoop.set_debug`. + :meth:`loop.set_debug`. * Set the log level of the :ref:`asyncio logger ` to :py:data:`logging.DEBUG`. For example, call ``logging.basicConfig(level=logging.DEBUG)`` at startup. @@ -35,11 +35,11 @@ Examples debug checks: * Log :ref:`coroutines defined but never "yielded from" ` -* :meth:`~AbstractEventLoop.call_soon` and :meth:`~AbstractEventLoop.call_at` methods +* :meth:`loop.call_soon` and :meth:`loop.call_at` methods raise an exception if they are called from the wrong thread. * Log the execution time of the selector * Log callbacks taking more than 100 ms to be executed. The - :attr:`AbstractEventLoop.slow_callback_duration` attribute is the minimum + :attr:`loop.slow_callback_duration` attribute is the minimum duration in seconds of "slow" callbacks. * :exc:`ResourceWarning` warnings are emitted when transports and event loops are :ref:`not closed explicitly `. @@ -51,7 +51,7 @@ Examples debug checks: .. seealso:: - The :meth:`AbstractEventLoop.set_debug` method and the :ref:`asyncio logger + The :meth:`loop.set_debug` method and the :ref:`asyncio logger `. @@ -75,7 +75,7 @@ For example, write:: Don't schedule directly a call to the :meth:`~Future.set_result` or the :meth:`~Future.set_exception` method of a future with -:meth:`AbstractEventLoop.call_soon`: the future can be cancelled before its method +:meth:`loop.call_soon`: the future can be cancelled before its method is called. If you wait for a future, you should check early if the future was cancelled to @@ -96,13 +96,14 @@ The :func:`shield` function can also be used to ignore cancellation. Concurrency and multithreading ------------------------------ -An event loop runs in a thread and executes all callbacks and tasks in the same -thread. While a task is running in the event loop, no other task is running in -the same thread. But when the task uses ``await``, the task is suspended -and the event loop executes the next task. +An event loop runs in a thread (typically the main thread) and executes +all callbacks and tasks in its thread. While a task is running in the +event loop, no other task is running in the same thread. When a task +executes an ``await`` expression, the task gets suspended and the +event loop executes the next task. To schedule a callback from a different thread, the -:meth:`AbstractEventLoop.call_soon_threadsafe` method should be used. Example:: +:meth:`loop.call_soon_threadsafe` method should be used. Example:: loop.call_soon_threadsafe(callback, *args) @@ -122,7 +123,7 @@ To schedule a coroutine object from a different thread, the future = asyncio.run_coroutine_threadsafe(coro_func(), loop) result = future.result(timeout) # Wait for the result with a timeout -The :meth:`AbstractEventLoop.run_in_executor` method can be used with a thread pool +The :meth:`loop.run_in_executor` method can be used with a thread pool executor to execute a callback in different thread to not block the thread of the event loop. @@ -151,7 +152,7 @@ APIs like :ref:`protocols `. An executor can be used to run a task in a different thread or even in a different process, to not block the thread of the event loop. See the -:meth:`AbstractEventLoop.run_in_executor` method. +:meth:`loop.run_in_executor` method. .. seealso:: @@ -182,7 +183,7 @@ Detect coroutine objects never scheduled ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`ensure_future` or to the :meth:`AbstractEventLoop.create_task` method, +:func:`ensure_future` or to the :meth:`loop.create_task` method, the execution of the coroutine object will never be scheduled which is probably a bug. :ref:`Enable the debug mode of asyncio ` to :ref:`log a warning ` to detect it. @@ -204,7 +205,7 @@ Output in debug mode:: test() The fix is to call the :func:`ensure_future` function or the -:meth:`AbstractEventLoop.create_task` method with the coroutine object. +:meth:`loop.create_task` method with the coroutine object. .. seealso:: @@ -279,7 +280,7 @@ coroutine in another coroutine and use classic try/except:: loop.run_forever() loop.close() -Another option is to use the :meth:`AbstractEventLoop.run_until_complete` +Another option is to use the :meth:`loop.run_until_complete` function:: task = asyncio.ensure_future(bug()) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 10068538c7a4..c200844385c1 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -1,103 +1,161 @@ .. currentmodule:: asyncio -.. _asyncio-event-loop: -Base Event Loop -=============== +========== +Event Loop +========== -**Source code:** :source:`Lib/asyncio/events.py` -The event loop is the central execution device provided by :mod:`asyncio`. -It provides multiple facilities, including: +.. rubric:: Preface -* Registering, executing and cancelling delayed calls (timeouts). +An event loop is the central component of every asyncio application. +Event loops run asynchronous tasks and callbacks, perform network +IO operations, run subprocesses, etc. -* Creating client and server :ref:`transports ` for various - kinds of communication. +In general, it is *not recommended* to use event loops directly at +the application-level asyncio code. They should only be accessed +in low-level code in libraries and frameworks. -* Launching subprocesses and the associated :ref:`transports - ` for communication with an external program. +High-level asyncio applications should not need to work with event +loops and should use the :func:`asyncio.run` function to initialize +and run asynchronous code. -* Delegating costly function calls to a pool of threads. -.. class:: BaseEventLoop +.. rubric:: Accessing Event Loop - This class is an implementation detail. It is a subclass of - :class:`AbstractEventLoop` and may be a base class of concrete - event loop implementations found in :mod:`asyncio`. It should not - be used directly; use :class:`AbstractEventLoop` instead. - ``BaseEventLoop`` should not be subclassed by third-party code; the - internal interface is not stable. +The following low-level functions can be used to get, set, or create +an event loop: -.. class:: AbstractEventLoop +.. function:: get_running_loop() - Abstract base class of event loops. + Return the running event loop in the current OS thread. - This class is :ref:`not thread safe `. + If there is no running event loop a :exc:`RuntimeError` is raised. + This function can only be called from a coroutine or a callback. -Run an event loop ------------------ + .. versionadded:: 3.7 -.. method:: AbstractEventLoop.run_forever() +.. function:: get_event_loop() - Run until :meth:`stop` is called. If :meth:`stop` is called before - :meth:`run_forever()` is called, this polls the I/O selector once - with a timeout of zero, runs all callbacks scheduled in response to - I/O events (and those that were already scheduled), and then exits. - If :meth:`stop` is called while :meth:`run_forever` is running, - this will run the current batch of callbacks and then exit. Note - that callbacks scheduled by callbacks will not run in that case; - they will run the next time :meth:`run_forever` is called. + Get the current event loop. If there is no current event loop set + in the current OS thread and :func:`set_event_loop` has not yet + been called, asyncio will create a new event loop and set it as the + current one. - .. versionchanged:: 3.5.1 + Because this function has a rather complex behavior (especially + when custom event loop policies are in use), it is recommended + to use the :func:`get_running_loop` function in coroutines and + callbacks instead. -.. method:: AbstractEventLoop.run_until_complete(future) + Consider also using the :func:`asyncio.run` function instead of + manually creating and closing an event loop. - Run until the :class:`Future` is done. +.. function:: set_event_loop(loop) - If the argument is a :ref:`coroutine object `, it is wrapped by - :func:`ensure_future`. + Set *loop* as a current event loop for the current OS thread. - Return the Future's result, or raise its exception. +.. function:: new_event_loop() -.. method:: AbstractEventLoop.is_running() + Create a new event loop object. - Returns running status of event loop. +Note that the behaviour of :func:`get_event_loop`, :func:`set_event_loop`, +and :func:`new_event_loop` functions can be altered by +:ref:`setting a custom event loop policy `. -.. method:: AbstractEventLoop.stop() - Stop running the event loop. +.. rubric:: Contents - This causes :meth:`run_forever` to exit at the next suitable - opportunity (see there for more details). +This documentation page contains the following sections: - .. versionchanged:: 3.5.1 +* The `Event Loop Methods`_ section is a reference documentation of + event loop APIs; + +* The `Callback Handles`_ section documents the :class:`Handle` and + :class:`TimerHandle`, instances of which are returned from functions + :meth:`loop.call_soon`, :meth:`loop.call_later`, etc; + +* The `Server Objects`_ sections documents types returned from + event loop methods like :meth:`loop.create_server`; + +* The `Event Loops Implementations`_ section documents the + :class:`SelectorEventLoop` and :class:`ProactorEventLoop` classes; + +* The `Examples`_ section showcases how to work with some event + loop APIs. + + +.. _asyncio-event-loop: + +Event Loop Methods +================== + +Event loops provide the following **low-level** APIs: + +.. contents:: + :depth: 1 + :local: + + +Running and stopping the loop +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. method:: loop.run_until_complete(future) + + Run until the *future* (an instance of :class:`Future`) is + completed. + + If the argument is a :ref:`coroutine object ` it + is implicitly wrapped into an :class:`asyncio.Task`. + + Return the Future's result or raise its exception. + +.. method:: loop.run_forever() + + Run the event loop until :meth:`stop` is called. + + If :meth:`stop` is called before :meth:`run_forever()` is called, + the loop will poll the I/O selector once with a timeout of zero, + run all callbacks scheduled in response to I/O events (and + those that were already scheduled), and then exit. + + If :meth:`stop` is called while :meth:`run_forever` is running, + the loop will run the current batch of callbacks and then exit. + Note that callbacks scheduled by callbacks will not run in that + case; they will run the next time :meth:`run_forever` or + :meth:`run_until_complete` is called. -.. method:: AbstractEventLoop.is_closed() +.. method:: loop.stop() - Returns ``True`` if the event loop was closed. + Stop the event loop. - .. versionadded:: 3.4.2 +.. method:: loop.is_running() -.. method:: AbstractEventLoop.close() + Return ``True`` if the event loop is currently running. - Close the event loop. The loop must not be running. Pending - callbacks will be lost. +.. method:: loop.is_closed() - This clears the queues and shuts down the executor, but does not wait for - the executor to finish. + Return ``True`` if the event loop was closed. - This is idempotent and irreversible. No other methods should be called after - this one. +.. method:: loop.close() + Close the event loop. -.. coroutinemethod:: AbstractEventLoop.shutdown_asyncgens() + The loop cannot not be running when this function is called. + Any pending callbacks will be discarded. + + This method clears all queues and shuts down the executor, but does + not wait for the executor to finish. + + This method is idempotent and irreversible. No other methods + should be called after the event loop is closed. + +.. coroutinemethod:: loop.shutdown_asyncgens() Schedule all currently open :term:`asynchronous generator` objects to close with an :meth:`~agen.aclose()` call. After calling this method, - the event loop will issue a warning whenever a new asynchronous generator - is iterated. Should be used to finalize all scheduled asynchronous - generators reliably. Example:: + the event loop will issue a warning if a new asynchronous generator + is iterated. Should be used to reliably finalize all scheduled + asynchronous generators, e.g.: try: loop.run_forever() @@ -110,163 +168,147 @@ Run an event loop .. _asyncio-pass-keywords: -Calls ------ - -Most :mod:`asyncio` functions don't accept keywords. If you want to pass -keywords to your callback, use :func:`functools.partial`. For example, -``loop.call_soon(functools.partial(print, "Hello", flush=True))`` will call -``print("Hello", flush=True)``. - -.. note:: - :func:`functools.partial` is better than ``lambda`` functions, because - :mod:`asyncio` can inspect :func:`functools.partial` object to display - parameters in debug mode, whereas ``lambda`` functions have a poor - representation. - -.. method:: AbstractEventLoop.call_soon(callback, *args, context=None) +Scheduling callbacks +^^^^^^^^^^^^^^^^^^^^ - Arrange for a callback to be called as soon as possible. The callback is - called after :meth:`call_soon` returns, when control returns to the event - loop. +.. method:: loop.call_soon(callback, *args, context=None) - This operates as a :abbr:`FIFO (first-in, first-out)` queue, callbacks - are called in the order in which they are registered. Each callback - will be called exactly once. + Schedule *callback* to be called with *args* arguments at + the next iteration of the event loop. - Any positional arguments after the callback will be passed to the - callback when it is called. + Callbacks are called in the order in which they are registered. + Each callback will be called exactly once. - An optional keyword-only *context* argument allows specifying a custom - :class:`contextvars.Context` for the *callback* to run in. The current - context is used when no *context* is provided. + An optional keyword-only *context* argument allows specifying a + custom :class:`contextvars.Context` for the *callback* to run in. + The current context is used when no *context* is provided. An instance of :class:`asyncio.Handle` is returned, which can be used to cancel the callback. - :ref:`Use functools.partial to pass keywords to the callback - `. +.. method:: loop.call_soon_threadsafe(callback, *args, context=None) - .. versionchanged:: 3.7 - The *context* keyword-only parameter was added. See :pep:`567` - for more details. - -.. method:: AbstractEventLoop.call_soon_threadsafe(callback, *args, context=None) - - Like :meth:`call_soon`, but thread safe. + A thread-safe variant of :meth:`call_soon`. Must be used to + schedule callbacks *from another thread*. See the :ref:`concurrency and multithreading ` section of the documentation. - .. versionchanged:: 3.7 - The *context* keyword-only parameter was added. See :pep:`567` - for more details. +.. versionchanged:: 3.7 + The *context* keyword-only parameter was added. See :pep:`567` + for more details. +.. note:: -.. _asyncio-delayed-calls: + Most :mod:`asyncio` scheduling functions don't allow to pass + keyword arguments. To do that, use :func:`functools.partial`, + e.g.:: -Delayed calls -------------- + # will schedule "print("Hello", flush=True)": + loop.call_soon( + functools.partial(print, "Hello", flush=True)) -The event loop has its own internal clock for computing timeouts. -Which clock is used depends on the (platform-specific) event loop -implementation; ideally it is a monotonic clock. This will generally be -a different clock than :func:`time.time`. + Using partial objects is usually more convenient than using lambdas, + as asyncio can better render partial objects in debug and error + messages. -.. method:: AbstractEventLoop.call_later(delay, callback, *args, context=None) +.. _asyncio-delayed-calls: - Arrange for the *callback* to be called after the given *delay* - seconds (either an int or float). +Scheduling delayed callbacks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - An instance of :class:`asyncio.TimerHandle` is returned, which can be - used to cancel the callback. +Event loop provides mechanisms to schedule callback functions +to be called at some point in the future. Event loop uses monotonic +clocks to track time. + + +.. method:: loop.call_later(delay, callback, *args, context=None) - *callback* will be called exactly once per call to :meth:`call_later`. - If two callbacks are scheduled for exactly the same time, it is - undefined which will be called first. + Schedule *callback* to be called after the given *delay* + number of seconds (can be either an int or a float). - The optional positional *args* will be passed to the callback when it - is called. If you want the callback to be called with some named - arguments, use a closure or :func:`functools.partial`. + An instance of :class:`asyncio.TimerHandle` is returned which can + be used to cancel the callback. - An optional keyword-only *context* argument allows specifying a custom - :class:`contextvars.Context` for the *callback* to run in. The current - context is used when no *context* is provided. + *callback* will be called exactly once. If two callbacks are + scheduled for exactly the same time, it is undefined which will + be called first. - :ref:`Use functools.partial to pass keywords to the callback - `. + The optional positional *args* will be passed to the callback when + it is called. If you want the callback to be called with keyword + arguments use :func:`functools.partial`. + + An optional keyword-only *context* argument allows specifying a + custom :class:`contextvars.Context` for the *callback* to run in. + The current context is used when no *context* is provided. .. versionchanged:: 3.7 The *context* keyword-only parameter was added. See :pep:`567` for more details. -.. method:: AbstractEventLoop.call_at(when, callback, *args, context=None) +.. method:: loop.call_at(when, callback, *args, context=None) - Arrange for the *callback* to be called at the given absolute timestamp - *when* (an int or float), using the same time reference as - :meth:`AbstractEventLoop.time`. + Schedule *callback* to be called at the given absolute timestamp + *when* (an int or a float), using the same time reference as + :meth:`loop.time`. This method's behavior is the same as :meth:`call_later`. - An instance of :class:`asyncio.TimerHandle` is returned, which can be - used to cancel the callback. - - :ref:`Use functools.partial to pass keywords to the callback - `. + An instance of :class:`asyncio.TimerHandle` is returned which can + be used to cancel the callback. .. versionchanged:: 3.7 The *context* keyword-only parameter was added. See :pep:`567` for more details. -.. method:: AbstractEventLoop.time() +.. method:: loop.time() - Return the current time, as a :class:`float` value, according to the - event loop's internal clock. + Return the current time, as a :class:`float` value, according to + the event loop's internal monotonic clock. + +.. note:: + + Timeouts (relative *delay* or absolute *when*) should not + exceed one day. .. seealso:: The :func:`asyncio.sleep` function. -Futures -------- +Creating Futures and Tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: AbstractEventLoop.create_future() +.. method:: loop.create_future() - Create an :class:`asyncio.Future` object attached to the loop. + Create an :class:`asyncio.Future` object attached to the event loop. - This is a preferred way to create futures in asyncio, as event - loop implementations can provide alternative implementations - of the Future class (with better performance or instrumentation). + This is the preferred way to create Futures in asyncio, that lets + third-party event loops to provide alternative implementations of + the Future object (with better performance or instrumentation). .. versionadded:: 3.5.2 +.. method:: loop.create_task(coro, \*, name=None) -Tasks ------ - -.. method:: AbstractEventLoop.create_task(coro, \*, name=None) + Schedule the execution of a :ref:`coroutine`. + Return a :class:`Task` object. - Schedule the execution of a :ref:`coroutine object `: wrap it in - a future. Return a :class:`Task` object. + Third-party event loops can use their own subclass of :class:`Task` + for interoperability. In this case, the result type is a subclass + of :class:`Task`. - Third-party event loops can use their own subclass of :class:`Task` for - interoperability. In this case, the result type is a subclass of - :class:`Task`. - - If the *name* argument is provided and not ``None``, it is set as the name - of the task using :meth:`Task.set_name`. - - .. versionadded:: 3.4.2 + If the *name* argument is provided and not ``None``, it is set as + the name of the task using :meth:`Task.set_name`. .. versionchanged:: 3.8 Added the ``name`` parameter. -.. method:: AbstractEventLoop.set_task_factory(factory) +.. method:: loop.set_task_factory(factory) Set a task factory that will be used by - :meth:`AbstractEventLoop.create_task`. + :meth:`loop.create_task`. If *factory* is ``None`` the default task factory will be set. @@ -275,53 +317,59 @@ Tasks event loop, *coro* will be a coroutine object. The callable must return an :class:`asyncio.Future` compatible object. - .. versionadded:: 3.4.4 +.. method:: loop.get_task_factory() + + Return a task factory or ``None`` if the default one is in use. -.. method:: AbstractEventLoop.get_task_factory() - Return a task factory, or ``None`` if the default one is in use. +Opening network connections +^^^^^^^^^^^^^^^^^^^^^^^^^^^ - .. versionadded:: 3.4.4 +.. coroutinemethod:: loop.create_connection(protocol_factory, \ + host=None, port=None, \*, ssl=None, \ + family=0, proto=0, flags=0, sock=None, \ + local_addr=None, server_hostname=None, \ + ssl_handshake_timeout=None) + Open a streaming transport connection to a given + address specified by *host* and *port*. -Creating connections --------------------- + The socket family can be either :py:data:`~socket.AF_INET` or + :py:data:`~socket.AF_INET6` depending on *host* (or the *family* + argument, if provided). -.. coroutinemethod:: AbstractEventLoop.create_connection(protocol_factory, host=None, port=None, \*, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None, ssl_handshake_timeout=None) + The socket type will be :py:data:`~socket.SOCK_STREAM`. - Create a streaming transport connection to a given Internet *host* and - *port*: socket family :py:data:`~socket.AF_INET` or - :py:data:`~socket.AF_INET6` depending on *host* (or *family* if specified), - socket type :py:data:`~socket.SOCK_STREAM`. *protocol_factory* must be a - callable returning a :ref:`protocol ` instance. + *protocol_factory* must be a callable returning an + :ref:`asyncio protocol ` implementation. This method will try to establish the connection in the background. When successful, it returns a ``(transport, protocol)`` pair. The chronological synopsis of the underlying operation is as follows: - #. The connection is established, and a :ref:`transport ` - is created to represent it. + #. The connection is established and a :ref:`transport ` + is created for it. - #. *protocol_factory* is called without arguments and must return a - :ref:`protocol ` instance. + #. *protocol_factory* is called without arguments and is expected to + return a :ref:`protocol ` instance. - #. The protocol instance is tied to the transport, and its - :meth:`connection_made` method is called. + #. The protocol instance is coupled with the transport by calling its + :meth:`~BaseProtocol.connection_made` method. - #. The coroutine returns successfully with the ``(transport, protocol)`` - pair. + #. A ``(transport, protocol)`` tuple is returned on success. - The created transport is an implementation-dependent bidirectional stream. + The created transport is an implementation-dependent bidirectional + stream. .. note:: *protocol_factory* can be any kind of callable, not necessarily a class. For example, if you want to use a pre-created protocol instance, you can pass ``lambda: my_protocol``. - Options that change how the connection is created: + Other arguments: - * *ssl*: if given and not false, a SSL/TLS transport is created + * *ssl*: if given and not false, an SSL/TLS transport is created (by default a plain TCP transport is created). If *ssl* is a :class:`ssl.SSLContext` object, this context is used to create the transport; if *ssl* is :const:`True`, a context with some @@ -359,28 +407,41 @@ Creating connections The *ssl_handshake_timeout* parameter. + .. versionchanged:: 3.6 + + The socket option :py:data:`~socket.TCP_NODELAY` is set by default + for all TCP connections. + .. versionchanged:: 3.5 - On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. + Added support for SSL/TLS for :class:`ProactorEventLoop`. .. seealso:: - The :func:`open_connection` function can be used to get a pair of - (:class:`StreamReader`, :class:`StreamWriter`) instead of a protocol. + The :func:`open_connection` function is a high-level alternative + API. It returns a pair of (:class:`StreamReader`, :class:`StreamWriter`) + that can be used directly in async/await code. +.. coroutinemethod:: loop.create_datagram_endpoint(protocol_factory, \ + local_addr=None, remote_addr=None, \*, \ + family=0, proto=0, flags=0, \ + reuse_address=None, reuse_port=None, \ + allow_broadcast=None, sock=None) -.. coroutinemethod:: AbstractEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0, reuse_address=None, reuse_port=None, allow_broadcast=None, sock=None) + Create a datagram connection. - Create datagram connection: socket family :py:data:`~socket.AF_INET`, - :py:data:`~socket.AF_INET6` or :py:data:`~socket.AF_UNIX` depending on - *host* (or *family* if specified), socket type - :py:data:`~socket.SOCK_DGRAM`. *protocol_factory* must be a - callable returning a :ref:`protocol ` instance. + The socket family can be either :py:data:`~socket.AF_INET`, + :py:data:`~socket.AF_INET6`, or :py:data:`~socket.AF_UNIX`, + depending on *host* (or the *family* argument, if provided). - This method will try to establish the connection in the background. - When successful, it returns a ``(transport, protocol)`` pair. + The socket type will be :py:data:`~socket.SOCK_DGRAM`. + + *protocol_factory* must be a callable returning a + :ref:`protocol ` implementation. + + A tuple of ``(transport, protocol)`` is returned on success. - Options changing how the connection is created: + Other arguments: * *local_addr*, if given, is a ``(local_host, local_port)`` tuple used to bind the socket to locally. The *local_host* and *local_port* @@ -396,7 +457,7 @@ Creating connections corresponding :mod:`socket` module constants. * *reuse_address* tells the kernel to reuse a local socket in - TIME_WAIT state, without waiting for its natural timeout to + ``TIME_WAIT`` state, without waiting for its natural timeout to expire. If not specified will automatically be set to ``True`` on UNIX. @@ -423,21 +484,24 @@ Creating connections The *family*, *proto*, *flags*, *reuse_address*, *reuse_port, *allow_broadcast*, and *sock* parameters were added. -.. coroutinemethod:: AbstractEventLoop.create_unix_connection(protocol_factory, path=None, \*, ssl=None, sock=None, server_hostname=None, ssl_handshake_timeout=None) +.. coroutinemethod:: loop.create_unix_connection(protocol_factory, \ + path=None, \*, ssl=None, sock=None, \ + server_hostname=None, ssl_handshake_timeout=None) - Create UNIX connection: socket family :py:data:`~socket.AF_UNIX`, socket - type :py:data:`~socket.SOCK_STREAM`. The :py:data:`~socket.AF_UNIX` socket - family is used to communicate between processes on the same machine - efficiently. + Create UNIX connection. - This method will try to establish the connection in the background. - When successful, it returns a ``(transport, protocol)`` pair. + The socket family will be :py:data:`~socket.AF_UNIX`; socket + type will be :py:data:`~socket.SOCK_STREAM`. - *path* is the name of a UNIX domain socket, and is required unless a *sock* - parameter is specified. Abstract UNIX sockets, :class:`str`, - :class:`bytes`, and :class:`~pathlib.Path` paths are supported. + A tuple of ``(transport, protocol)`` is returned on success. - See the :meth:`AbstractEventLoop.create_connection` method for parameters. + *path* is the name of a UNIX domain socket and is required, + unless a *sock* parameter is specified. Abstract UNIX sockets, + :class:`str`, :class:`bytes`, and :class:`~pathlib.Path` paths are + supported. + + See the documentation of the :meth:`loop.create_connection` method + for information about arguments to this method. Availability: UNIX. @@ -450,19 +514,23 @@ Creating connections The *path* parameter can now be a :term:`path-like object`. -Creating listening connections ------------------------------- +Creating network servers +^^^^^^^^^^^^^^^^^^^^^^^^ -.. coroutinemethod:: AbstractEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, start_serving=True) +.. coroutinemethod:: loop.create_server(protocol_factory, \ + host=None, port=None, \*, \ + family=socket.AF_UNSPEC, \ + flags=socket.AI_PASSIVE, \ + sock=None, backlog=100, ssl=None, \ + reuse_address=None, reuse_port=None, \ + ssl_handshake_timeout=None, start_serving=True) - Create a TCP server (socket type :data:`~socket.SOCK_STREAM`) bound to - *host* and *port*. + Create a TCP server (socket type :data:`~socket.SOCK_STREAM`) listening + on the *host* and *port* address. - Return a :class:`Server` object, its :attr:`~Server.sockets` attribute - contains created sockets. Use the :meth:`Server.close` method to stop the - server: close listening sockets. + Returns a :class:`Server` object. - Parameters: + Arguments: * The *host* parameter can be a string, in that case the TCP server is bound to *host* and *port*. The *host* parameter can also be a sequence @@ -472,8 +540,9 @@ Creating listening connections for IPv4 and another one for IPv6). * *family* can be set to either :data:`socket.AF_INET` or - :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. If not set - it will be determined from host (defaults to :data:`socket.AF_UNSPEC`). + :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. + If not set it will be determined from host name + (defaults to :data:`~socket.AF_UNSPEC`). * *flags* is a bitmask for :meth:`getaddrinfo`. @@ -488,7 +557,7 @@ Creating listening connections accepted connections. * *reuse_address* tells the kernel to reuse a local socket in - TIME_WAIT state, without waiting for its natural timeout to + ``TIME_WAIT`` state, without waiting for its natural timeout to expire. If not specified will automatically be set to ``True`` on UNIX. @@ -509,30 +578,40 @@ Creating listening connections .. versionadded:: 3.7 - *ssl_handshake_timeout* and *start_serving* parameters. + Added *ssl_handshake_timeout* and *start_serving* parameters. - .. versionchanged:: 3.5 + .. versionchanged:: 3.6 - On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. + The socket option :py:data:`~socket.TCP_NODELAY` is set by default + for all TCP connections. - .. seealso:: + .. versionchanged:: 3.5 - The function :func:`start_server` creates a (:class:`StreamReader`, - :class:`StreamWriter`) pair and calls back a function with this pair. + Added support for SSL/TLS on Windows with + :class:`ProactorEventLoop`. .. versionchanged:: 3.5.1 - The *host* parameter can now be a sequence of strings. + The *host* parameter can be a sequence of strings. + + .. seealso:: + + The :func:`start_server` function is a higher-level alternative API + that returns a pair of :class:`StreamReader` and :class:`StreamWriter` + that can be used in an async/await code. -.. coroutinemethod:: AbstractEventLoop.create_unix_server(protocol_factory, path=None, \*, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, start_serving=True) +.. coroutinemethod:: loop.create_unix_server(protocol_factory, path=None, \ + \*, sock=None, backlog=100, ssl=None, \ + ssl_handshake_timeout=None, start_serving=True) - Similar to :meth:`AbstractEventLoop.create_server`, but specific to the - socket family :py:data:`~socket.AF_UNIX`. + Similar to :meth:`loop.create_server` but works with the + :py:data:`~socket.AF_UNIX` socket family. - *path* is the name of a UNIX domain socket, and is required unless a *sock* - parameter is specified. Abstract UNIX sockets, :class:`str`, - :class:`bytes`, and :class:`~pathlib.Path` paths are supported. + *path* is the name of a UNIX domain socket, and is required, + unless a *sock* argument is provided. Abstract UNIX sockets, + :class:`str`, :class:`bytes`, and :class:`~pathlib.Path` paths + are supported. Availability: UNIX. @@ -544,26 +623,27 @@ Creating listening connections The *path* parameter can now be a :class:`~pathlib.Path` object. -.. coroutinemethod:: BaseEventLoop.connect_accepted_socket(protocol_factory, sock, \*, ssl=None, ssl_handshake_timeout=None) +.. coroutinemethod:: loop.connect_accepted_socket(protocol_factory, \ + sock, \*, ssl=None, ssl_handshake_timeout=None) - Handle an accepted connection. + Wrap an already accepted connection into a transport/protocol pair. - This is used by servers that accept connections outside of - asyncio but that use asyncio to handle them. + This method can be used by servers that accept connections outside + of asyncio but that use asyncio to handle them. Parameters: - * *sock* is a preexisting socket object returned from an ``accept`` - call. + * *sock* is a preexisting socket object returned from + :meth:`socket.accept `. - * *ssl* can be set to an :class:`~ssl.SSLContext` to enable SSL over the - accepted connections. + * *ssl* can be set to an :class:`~ssl.SSLContext` to enable SSL over + the accepted connections. * *ssl_handshake_timeout* is (for an SSL connection) the time in seconds to wait for the SSL handshake to complete before aborting the connection. ``60.0`` seconds if ``None`` (default). - When completed it returns a ``(transport, protocol)`` pair. + Returns a ``(transport, protocol)`` pair. .. versionadded:: 3.7 @@ -572,15 +652,14 @@ Creating listening connections .. versionadded:: 3.5.3 -File Transferring ------------------ +Transferring files +^^^^^^^^^^^^^^^^^^ -.. coroutinemethod:: AbstractEventLoop.sendfile(transport, file, \ - offset=0, count=None, \ - *, fallback=True) +.. coroutinemethod:: loop.sendfile(transport, file, \ + offset=0, count=None, *, fallback=True) - Send a *file* to *transport*, return the total number of bytes - which were sent. + Send a *file* over a *transport*. Return the total number of bytes + sent. The method uses high-performance :meth:`os.sendfile` if available. @@ -594,7 +673,7 @@ File Transferring which were sent. *fallback* set to ``True`` makes asyncio to manually read and send - the file when the platform does not support the sendfile syscall + the file when the platform does not support the sendfile system call (e.g. Windows or SSL socket on Unix). Raise :exc:`SendfileNotAvailableError` if the system does not support @@ -604,26 +683,28 @@ File Transferring TLS Upgrade ------------ +^^^^^^^^^^^ -.. coroutinemethod:: AbstractEventLoop.start_tls(transport, protocol, sslcontext, \*, server_side=False, server_hostname=None, ssl_handshake_timeout=None) +.. coroutinemethod:: loop.start_tls(transport, protocol, \ + sslcontext, \*, server_side=False, \ + server_hostname=None, ssl_handshake_timeout=None) - Upgrades an existing connection to TLS. + Upgrade an existing transport-based connection to TLS. - Returns a new transport instance, that the *protocol* must start using + Return a new transport instance, that the *protocol* must start using immediately after the *await*. The *transport* instance passed to the *start_tls* method should never be used again. Parameters: * *transport* and *protocol* instances that methods like - :meth:`~AbstractEventLoop.create_server` and - :meth:`~AbstractEventLoop.create_connection` return. + :meth:`~loop.create_server` and + :meth:`~loop.create_connection` return. * *sslcontext*: a configured instance of :class:`~ssl.SSLContext`. * *server_side* pass ``True`` when a server-side connection is being - upgraded (like the one created by :meth:`~AbstractEventLoop.create_server`). + upgraded (like the one created by :meth:`~loop.create_server`). * *server_hostname*: sets or overrides the host name that the target server's certificate will be matched against. @@ -635,120 +716,112 @@ TLS Upgrade .. versionadded:: 3.7 -Watch file descriptors ----------------------- - -On Windows with :class:`SelectorEventLoop`, only socket handles are supported -(ex: pipe file descriptors are not supported). - -On Windows with :class:`ProactorEventLoop`, these methods are not supported. - -.. method:: AbstractEventLoop.add_reader(fd, callback, \*args) +Watching file descriptors +^^^^^^^^^^^^^^^^^^^^^^^^^ - Start watching the file descriptor for read availability and then call the - *callback* with specified arguments. +.. method:: loop.add_reader(fd, callback, \*args) - :ref:`Use functools.partial to pass keywords to the callback - `. + Start watching the file descriptor for read availability and + call the *callback* with specified arguments. -.. method:: AbstractEventLoop.remove_reader(fd) +.. method:: loop.remove_reader(fd) Stop watching the file descriptor for read availability. -.. method:: AbstractEventLoop.add_writer(fd, callback, \*args) +.. method:: loop.add_writer(fd, callback, \*args) - Start watching the file descriptor for write availability and then call the - *callback* with specified arguments. + Start watching the file descriptor for write availability and then + call the *callback* with specified arguments. - :ref:`Use functools.partial to pass keywords to the callback - `. + Use :func:`functools.partial` :ref:`to pass keywords + ` to *func*. -.. method:: AbstractEventLoop.remove_writer(fd) +.. method:: loop.remove_writer(fd) Stop watching the file descriptor for write availability. -The :ref:`watch a file descriptor for read events ` -example uses the low-level :meth:`AbstractEventLoop.add_reader` method to register -the file descriptor of a socket. +See also :ref:`Platform Support ` section +for some limitations of these methods. -Low-level socket operations ---------------------------- +Working with socket objects directly +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. coroutinemethod:: AbstractEventLoop.sock_recv(sock, nbytes) +In general, protocols implementations that use transport-based APIs +such as :meth:`loop.create_connection` and :meth:`loop.create_server` +are faster than implementations that work with sockets directly. +However, there are use cases when performance is not critical and +working with :class:`~socket.socket` objects directly is more +convenient. - Receive data from the socket. Modeled after blocking - :meth:`socket.socket.recv` method. +.. coroutinemethod:: loop.sock_recv(sock, nbytes) - The return value is a bytes object - representing the data received. The maximum amount of data to be received - at once is specified by *nbytes*. + Receive data. Asynchronous version of + :meth:`socket.recv() `. - With :class:`SelectorEventLoop` event loop, the socket *sock* must be - non-blocking. + The received data is returned as a bytes object. The maximum amount + of data to be received is specified by the *nbytes* argument. + + The socket *sock* must be non-blocking. .. versionchanged:: 3.7 Even though the method was always documented as a coroutine method, before Python 3.7 it returned a :class:`Future`. - Since Python 3.7, this is an ``async def`` method. + Since Python 3.7 this is an ``async def`` method. -.. coroutinemethod:: AbstractEventLoop.sock_recv_into(sock, buf) +.. coroutinemethod:: loop.sock_recv_into(sock, buf) - Receive data from the socket. Modeled after blocking - :meth:`socket.socket.recv_into` method. + Receive data into a buffer. Modeled after the blocking + :meth:`socket.recv_into() ` method. - The received data is written into *buf* (a writable buffer). - The return value is the number of bytes written. + Return the number of bytes written to the buffer. - With :class:`SelectorEventLoop` event loop, the socket *sock* must be - non-blocking. + The socket *sock* must be non-blocking. .. versionadded:: 3.7 -.. coroutinemethod:: AbstractEventLoop.sock_sendall(sock, data) +.. coroutinemethod:: loop.sock_sendall(sock, data) - Send data to the socket. Modeled after blocking - :meth:`socket.socket.sendall` method. + Send data to the socket. Asynchronous version of + :meth:`socket.sendall() `. - The socket must be connected to a remote socket. This method continues to send data from *data* until either all data has been sent or an error occurs. ``None`` is returned on success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully processed by the receiving end of the connection. - With :class:`SelectorEventLoop` event loop, the socket *sock* must be - non-blocking. + The socket *sock* must be non-blocking. .. versionchanged:: 3.7 Even though the method was always documented as a coroutine method, before Python 3.7 it returned an :class:`Future`. Since Python 3.7, this is an ``async def`` method. -.. coroutinemethod:: AbstractEventLoop.sock_connect(sock, address) +.. coroutinemethod:: loop.sock_connect(sock, address) + + Connect to a remote socket at *address*. - Connect to a remote socket at *address*. Modeled after - blocking :meth:`socket.socket.connect` method. + Asynchronous version of :meth:`socket.connect() `. - With :class:`SelectorEventLoop` event loop, the socket *sock* must be - non-blocking. + The socket *sock* must be non-blocking. .. versionchanged:: 3.5.2 ``address`` no longer needs to be resolved. ``sock_connect`` will try to check if the *address* is already resolved by calling :func:`socket.inet_pton`. If not, - :meth:`AbstractEventLoop.getaddrinfo` will be used to resolve the + :meth:`loop.getaddrinfo` will be used to resolve the *address*. .. seealso:: - :meth:`AbstractEventLoop.create_connection` + :meth:`loop.create_connection` and :func:`asyncio.open_connection() `. -.. coroutinemethod:: AbstractEventLoop.sock_accept(sock) +.. coroutinemethod:: loop.sock_accept(sock) - Accept a connection. Modeled after blocking - :meth:`socket.socket.accept`. + Accept a connection. Modeled after the blocking + :meth:`socket.accept() ` method. The socket must be bound to an address and listening for connections. The return value is a pair ``(conn, address)`` where *conn* @@ -765,16 +838,15 @@ Low-level socket operations .. seealso:: - :meth:`AbstractEventLoop.create_server` and :func:`start_server`. + :meth:`loop.create_server` and :func:`start_server`. -.. coroutinemethod:: AbstractEventLoop.sock_sendfile(sock, file, \ - offset=0, count=None, \ - *, fallback=True) +.. coroutinemethod:: loop.sock_sendfile(sock, file, offset=0, count=None, \ + \*, fallback=True) - Send a file using high-performance :mod:`os.sendfile` if possible - and return the total number of bytes which were sent. + Send a file using high-performance :mod:`os.sendfile` if possible. + Return the total number of bytes which were sent. - Asynchronous version of :meth:`socket.socket.sendfile`. + Asynchronous version of :meth:`socket.sendfile() `. *sock* must be non-blocking :class:`~socket.socket` of :const:`socket.SOCK_STREAM` type. @@ -795,21 +867,22 @@ Low-level socket operations Raise :exc:`SendfileNotAvailableError` if the system does not support *sendfile* syscall and *fallback* is ``False``. + The socket *sock* must be non-blocking. + .. versionadded:: 3.7 -Resolve host name ------------------ +DNS +^^^ -.. coroutinemethod:: AbstractEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) +.. coroutinemethod:: loop.getaddrinfo(host, port, \*, family=0, \ + type=0, proto=0, flags=0) - This method is a :ref:`coroutine `, similar to - :meth:`socket.getaddrinfo` function but non-blocking. + Asynchronous version of :meth:`socket.getaddrinfo`. -.. coroutinemethod:: AbstractEventLoop.getnameinfo(sockaddr, flags=0) +.. coroutinemethod:: loop.getnameinfo(sockaddr, flags=0) - This method is a :ref:`coroutine `, similar to - :meth:`socket.getnameinfo` function but non-blocking. + Asynchronous version of :meth:`socket.getnameinfo`. .. versionchanged:: 3.7 Both *getaddrinfo* and *getnameinfo* methods were always documented @@ -818,95 +891,99 @@ Resolve host name both methods are coroutines. -Connect pipes -------------- +Working with pipes +^^^^^^^^^^^^^^^^^^ -On Windows with :class:`SelectorEventLoop`, these methods are not supported. -Use :class:`ProactorEventLoop` to support pipes on Windows. +.. coroutinemethod:: loop.connect_read_pipe(protocol_factory, pipe) -.. coroutinemethod:: AbstractEventLoop.connect_read_pipe(protocol_factory, pipe) + Register a read-pipe in the event loop. - Register read pipe in eventloop. + *protocol_factory* must be a callable returning an + :ref:`asyncio protocol ` implementation. - *protocol_factory* should instantiate object with :class:`Protocol` - interface. *pipe* is a :term:`file-like object `. - Return pair ``(transport, protocol)``, where *transport* supports the - :class:`ReadTransport` interface. + *pipe* is a :term:`file-like object `. + + Return pair ``(transport, protocol)``, where *transport* supports + the :class:`ReadTransport` interface. With :class:`SelectorEventLoop` event loop, the *pipe* is set to non-blocking mode. -.. coroutinemethod:: AbstractEventLoop.connect_write_pipe(protocol_factory, pipe) +.. coroutinemethod:: loop.connect_write_pipe(protocol_factory, pipe) + + Register a write-pipe in the event loop. + + *protocol_factory* must be a callable returning an + :ref:`asyncio protocol ` implementation. - Register write pipe in eventloop. + *pipe* is :term:`file-like object `. - *protocol_factory* should instantiate object with :class:`BaseProtocol` - interface. *pipe* is :term:`file-like object `. Return pair ``(transport, protocol)``, where *transport* supports :class:`WriteTransport` interface. With :class:`SelectorEventLoop` event loop, the *pipe* is set to non-blocking mode. +.. note:: + + :class:`SelectorEventLoop` does not support the above methods on + Windows. Use :class:`ProactorEventLoop` instead. + .. seealso:: - The :meth:`AbstractEventLoop.subprocess_exec` and - :meth:`AbstractEventLoop.subprocess_shell` methods. + The :meth:`loop.subprocess_exec` and + :meth:`loop.subprocess_shell` methods. UNIX signals ------------- +^^^^^^^^^^^^ -Availability: UNIX only. - -.. method:: AbstractEventLoop.add_signal_handler(signum, callback, \*args) +.. method:: loop.add_signal_handler(signum, callback, \*args) Add a handler for a signal. Raise :exc:`ValueError` if the signal number is invalid or uncatchable. Raise :exc:`RuntimeError` if there is a problem setting up the handler. - :ref:`Use functools.partial to pass keywords to the callback - `. + Use :func:`functools.partial` :ref:`to pass keywords + ` to *func*. -.. method:: AbstractEventLoop.remove_signal_handler(sig) +.. method:: loop.remove_signal_handler(sig) Remove a handler for a signal. Return ``True`` if a signal handler was removed, ``False`` if not. +Availability: UNIX. + .. seealso:: The :mod:`signal` module. -Executor --------- - -Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or -pool of processes). By default, an event loop uses a thread pool executor -(:class:`~concurrent.futures.ThreadPoolExecutor`). +Executing code in thread or process pools +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: AbstractEventLoop.run_in_executor(executor, func, \*args) +.. method:: loop.run_in_executor(executor, func, \*args) Arrange for a *func* to be called in the specified executor. - The *executor* argument should be an :class:`~concurrent.futures.Executor` + The *executor* argument should be an :class:`concurrent.futures.Executor` instance. The default executor is used if *executor* is ``None``. - :ref:`Use functools.partial to pass keywords to the *func* - `. + Use :func:`functools.partial` :ref:`to pass keywords + ` to *func*. This method returns a :class:`asyncio.Future` object. .. versionchanged:: 3.5.3 - :meth:`BaseEventLoop.run_in_executor` no longer configures the + :meth:`loop.run_in_executor` no longer configures the ``max_workers`` of the thread pool executor it creates, instead leaving it up to the thread pool executor (:class:`~concurrent.futures.ThreadPoolExecutor`) to set the default. -.. method:: AbstractEventLoop.set_default_executor(executor) +.. method:: loop.set_default_executor(executor) Set *executor* as the default executor used by :meth:`run_in_executor`. *executor* should be an instance of @@ -917,13 +994,16 @@ pool of processes). By default, an event loop uses a thread pool executor :class:`~concurrent.futures.ThreadPoolExecutor` is deprecated and will trigger an error in Python 3.9. + *executor* must be an instance of + :class:`concurrent.futures.ThreadPoolExecutor`. + Error Handling API ------------------- +^^^^^^^^^^^^^^^^^^ Allows customizing how exceptions are handled in the event loop. -.. method:: AbstractEventLoop.set_exception_handler(handler) +.. method:: loop.set_exception_handler(handler) Set *handler* as the new event loop exception handler. @@ -936,14 +1016,14 @@ Allows customizing how exceptions are handled in the event loop. will be a ``dict`` object (see :meth:`call_exception_handler` documentation for details about context). -.. method:: AbstractEventLoop.get_exception_handler() +.. method:: loop.get_exception_handler() Return the exception handler, or ``None`` if the default one is in use. .. versionadded:: 3.5.2 -.. method:: AbstractEventLoop.default_exception_handler(context) +.. method:: loop.default_exception_handler(context) Default exception handler. @@ -954,7 +1034,7 @@ Allows customizing how exceptions are handled in the event loop. *context* parameter has the same meaning as in :meth:`call_exception_handler`. -.. method:: AbstractEventLoop.call_exception_handler(context) +.. method:: loop.call_exception_handler(context) Call the current event loop exception handler. @@ -975,10 +1055,10 @@ Allows customizing how exceptions are handled in the event loop. event loops. For any custom exception handling, use :meth:`set_exception_handler()` method. -Debug mode ----------- +Enabling debug mode +^^^^^^^^^^^^^^^^^^^ -.. method:: AbstractEventLoop.get_debug() +.. method:: loop.get_debug() Get the debug mode (:class:`bool`) of the event loop. @@ -986,29 +1066,167 @@ Debug mode :envvar:`PYTHONASYNCIODEBUG` is set to a non-empty string, ``False`` otherwise. - .. versionadded:: 3.4.2 - -.. method:: AbstractEventLoop.set_debug(enabled: bool) +.. method:: loop.set_debug(enabled: bool) Set the debug mode of the event loop. - .. versionadded:: 3.4.2 - .. seealso:: The :ref:`debug mode of asyncio `. -Server ------- -.. class:: Server +Running Subprocesses +^^^^^^^^^^^^^^^^^^^^ + +Methods described in this subsections are low-level. In an +async/await code consider using high-level convenient +:func:`asyncio.create_subprocess_shell` and +:func:`asyncio.create_subprocess_exec` functions instead. + +.. note:: + + The default event loop that asyncio is pre-configured + to use on **Windows** does not support subprocesses. + See :ref:`Subprocess Support on Windows ` + for details. + +.. coroutinemethod:: loop.subprocess_exec(protocol_factory, \*args, \ + stdin=subprocess.PIPE, stdout=subprocess.PIPE, \ + stderr=subprocess.PIPE, \*\*kwargs) + + Create a subprocess from one or more string arguments specified by + *args*. + + *args* must be a list of strings represented by: + + * :class:`str`; + * or :class:`bytes`, encoded to the + :ref:`filesystem encoding `. + + The first string specifies the program to execute, + and the remaining strings specify the arguments. Together string + arguments form the ``argv`` of the program. + + This is similar to the standard library :class:`subprocess.Popen` + class called with ``shell=False`` and the list of strings passed as + the first argument; however, where :class:`~subprocess.Popen` takes + a single argument which is list of strings, *subprocess_exec* + takes multiple string arguments. + + The *protocol_factory* must instantiate a subclass of the + :class:`asyncio.SubprocessProtocol` class. + + Other parameters: + + * *stdin*: either a file-like object representing a pipe to be + connected to the subprocess's standard input stream using + :meth:`~loop.connect_write_pipe`, or the + :const:`subprocess.PIPE` constant (default). By default a new + pipe will be created and connected. - Server listening on sockets. + * *stdout*: either a file-like object representing the pipe to be + connected to the subprocess's standard output stream using + :meth:`~loop.connect_read_pipe`, or the + :const:`subprocess.PIPE` constant (default). By default a new pipe + will be created and connected. + + * *stderr*: either a file-like object representing the pipe to be + connected to the subprocess's standard error stream using + :meth:`~loop.connect_read_pipe`, or one of + :const:`subprocess.PIPE` (default) or :const:`subprocess.STDOUT` + constants. + + By default a new pipe will be created and connected. When + :const:`subprocess.STDOUT` is specified, the subprocess' standard + error stream will be connected to the same pipe as the standard + output stream. + + * All other keyword arguments are passed to :class:`subprocess.Popen` + without interpretation, except for *bufsize*, *universal_newlines* + and *shell*, which should not be specified at all. + + See the constructor of the :class:`subprocess.Popen` class + for documentation on other arguments. + + Returns a pair of ``(transport, protocol)``, where *transport* + conforms to the :class:`asyncio.SubprocessTransport` base class. + +.. coroutinemethod:: loop.subprocess_shell(protocol_factory, cmd, \*, \ + stdin=subprocess.PIPE, stdout=subprocess.PIPE, \ + stderr=subprocess.PIPE, \*\*kwargs) + + Create a subprocess from *cmd*, which can be a :class:`str` or a + :class:`bytes` string encoded to the + :ref:`filesystem encoding `, + using the platform's "shell" syntax. + + This is similar to the standard library :class:`subprocess.Popen` + class called with ``shell=True``. + + The *protocol_factory* must instantiate a subclass of the + :class:`SubprocessProtocol` class. + + See :meth:`~loop.subprocess_exec` for more details about + the remaining arguments. + + Returns a pair of ``(transport, protocol)``, where *transport* + conforms to the :class:`SubprocessTransport` base class. + +.. note:: + It is the application's responsibility to ensure that all whitespace + and metacharacters are quoted appropriately to avoid `shell injection + `_ + vulnerabilities. The :func:`shlex.quote` function can be used to + properly escape whitespace and shell metacharacters in strings that + are going to be used to construct shell commands. - Object created by :meth:`AbstractEventLoop.create_server`, - :meth:`AbstractEventLoop.create_unix_server`, :func:`start_server`, - and :func:`start_unix_server` functions. Don't instantiate the class - directly. + +Callback Handles +================ + +.. class:: Handle + + A callback wrapper object returned by :meth:`loop.call_soon`, + :meth:`loop.call_soon_threadsafe`. + + .. method:: cancel() + + Cancel the call. If the callback is already canceled or executed, + this method has no effect. + + .. method:: cancelled() + + Return ``True`` if the call was cancelled. + + .. versionadded:: 3.7 + +.. class:: TimerHandle + + A callback wrapper object returned by :meth:`loop.call_later`, + and :meth:`loop.call_at`. + + The class is inherited from :class:`Handle`. + + .. method:: when() + + Return a scheduled callback time as :class:`float` seconds. + + The time is an absolute timestamp, using the same time + reference as :meth:`loop.time`. + + .. versionadded:: 3.7 + + +Server Objects +============== + +Server objects are created by :meth:`loop.create_server`, +:meth:`loop.create_unix_server`, :func:`start_server`, +and :func:`start_unix_server` functions. + +Do not instantiate the class directly. + +.. class:: Server *Server* objects are asynchronous context managers. When used in an ``async with`` statement, it's guaranteed that the Server object is @@ -1020,7 +1238,8 @@ Server async with srv: # some code - # At this point, srv is closed and no longer accepts new connections. + # At this point, srv is closed and no longer accepts new + connections. .. versionchanged:: 3.7 @@ -1031,8 +1250,8 @@ Server Stop serving: close listening sockets and set the :attr:`sockets` attribute to ``None``. - The sockets that represent existing incoming client connections are left - open. + The sockets that represent existing incoming client connections + are left open. The server is closed asynchronously, use the :meth:`wait_closed` coroutine to wait until the server is closed. @@ -1051,7 +1270,7 @@ Server the server is already being serving. The new *start_serving* keyword-only parameter to - :meth:`AbstractEventLoop.create_server` and + :meth:`loop.create_server` and :meth:`asyncio.start_server` allows to create a Server object that is not accepting connections right away. In which case this method, or :meth:`Server.serve_forever` can be used @@ -1097,8 +1316,8 @@ Server .. attribute:: sockets - List of :class:`socket.socket` objects the server is listening to, or - ``None`` if the server is closed. + List of :class:`socket.socket` objects the server is listening to, + or ``None`` if the server is closed. .. versionchanged:: 3.7 Prior to Python 3.7 ``Server.sockets`` used to return the @@ -1106,65 +1325,85 @@ Server of that list is returned. -Handle ------- +.. _asyncio-event-loops: -.. class:: Handle +Event Loops Implementations +=========================== - A callback wrapper object returned by :func:`AbstractEventLoop.call_soon`, - :func:`AbstractEventLoop.call_soon_threadsafe`. +asyncio ships with two different event loop implementations: +:class:`SelectorEventLoop` and :class:`ProactorEventLoop`. - .. method:: cancel() +By default asyncio is configured to use :class:`SelectorEventLoop` +on all platforms. - Cancel the call. If the callback is already canceled or executed, - this method has no effect. - .. method:: cancelled() +.. class:: SelectorEventLoop - Return ``True`` if the call was cancelled. + An event loop based on the :mod:`selectors` module. - .. versionadded:: 3.7 + Uses the most efficient *selector* available for the given + platform. It is also possible to manually configure what + exact selector implementation should be used:: -.. class:: TimerHandle + import asyncio + import selectors - A callback wrapper object returned by :func:`AbstractEventLoop.call_later`, - and :func:`AbstractEventLoop.call_at`. + selector = selectors.SelectSelector() + loop = asyncio.SelectorEventLoop(selector) + asyncio.set_event_loop(loop) - The class is inherited from :class:`Handle`. - .. method:: when() + Availability: UNIX, Windows. - Return a scheduled callback time as :class:`float` seconds. - The time is an absolute timestamp, using the same time - reference as :meth:`AbstractEventLoop.time`. +.. class:: ProactorEventLoop - .. versionadded:: 3.7 + An event loop for Windows that uses "I/O Completion Ports" (IOCP). + + Availability: Windows. + + An example how to use :class:`ProactorEventLoop` on Windows:: + + import asyncio + import sys + + if sys.platform == 'win32': + loop = asyncio.ProactorEventLoop() + asyncio.set_event_loop(loop) + + .. seealso:: + + `MSDN documentation on I/O Completion Ports + `_. -SendfileNotAvailableError -------------------------- +.. class:: AbstractEventLoop + Abstract base class for asyncio-compliant event loops. -.. exception:: SendfileNotAvailableError + The :ref:`Event Loop Methods ` section lists all + methods that an alternative implementation of ``AbstractEventLoop`` + should have defined. - Sendfile syscall is not available, subclass of :exc:`RuntimeError`. - Raised if the OS does not support sendfile syscall for - given socket or file type. +Examples +======== +Note that all examples in this section **purposefully** show how +to use low-level event loop APIs such as :meth:`loop.run_forever` +and :meth:`loop.call_soon`. Modern asyncio applications rarely +need to be written this way; consider using high-level functions +like :func:`asyncio.run`. -Event loop examples -------------------- .. _asyncio-hello-world-callback: Hello World with call_soon() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Example using the :meth:`AbstractEventLoop.call_soon` method to schedule a -callback. The callback displays ``"Hello World"`` and then stops the event -loop:: +An example using the :meth:`loop.call_soon` method to schedule a +callback. The callback displays ``"Hello World"`` and then stops the +event loop:: import asyncio @@ -1178,13 +1417,15 @@ loop:: loop.call_soon(hello_world, loop) # Blocking call interrupted by loop.stop() - loop.run_forever() - loop.close() + try: + loop.run_forever() + finally: + loop.close() .. seealso:: - The :ref:`Hello World coroutine ` example - uses a :ref:`coroutine `. + A similar :ref:`Hello World ` + example created with a coroutine and the :func:`run` function. .. _asyncio-date-callback: @@ -1192,9 +1433,9 @@ loop:: Display the current date with call_later() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Example of callback displaying the current date every second. The callback uses -the :meth:`AbstractEventLoop.call_later` method to reschedule itself during 5 -seconds, and then stops the event loop:: +An example of callback displaying the current date every second. The +callback uses the :meth:`loop.call_later` method to reschedule itself +during 5 seconds, and then stops the event loop:: import asyncio import datetime @@ -1213,14 +1454,15 @@ seconds, and then stops the event loop:: loop.call_soon(display_date, end_time, loop) # Blocking call interrupted by loop.stop() - loop.run_forever() - loop.close() + try: + loop.run_forever() + finally: + loop.close() .. seealso:: - The :ref:`coroutine displaying the current date - ` example uses a :ref:`coroutine - `. + A similar :ref:`current date ` example + created with a coroutine and the :func:`run` function. .. _asyncio-watch-read-event: @@ -1229,7 +1471,7 @@ Watch a file descriptor for read events ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Wait until a file descriptor received some data using the -:meth:`AbstractEventLoop.add_reader` method and then close the event loop:: +:meth:`loop.add_reader` method and then close the event loop:: import asyncio from socket import socketpair @@ -1241,8 +1483,10 @@ Wait until a file descriptor received some data using the def reader(): data = rsock.recv(100) print("Received:", data.decode()) + # We are done: unregister the file descriptor loop.remove_reader(rsock) + # Stop the event loop loop.stop() @@ -1252,30 +1496,33 @@ Wait until a file descriptor received some data using the # Simulate the reception of data from the network loop.call_soon(wsock.send, 'abc'.encode()) - # Run the event loop - loop.run_forever() - - # We are done, close sockets and the event loop - rsock.close() - wsock.close() - loop.close() + try: + # Run the event loop + loop.run_forever() + finally: + # We are done, close sockets and the event loop + rsock.close() + wsock.close() + loop.close() .. seealso:: - The :ref:`register an open socket to wait for data using a protocol - ` example uses a low-level protocol created by the - :meth:`AbstractEventLoop.create_connection` method. + * A similar :ref:`example ` + using transports, protocols, and the + :meth:`loop.create_connection` method. - The :ref:`register an open socket to wait for data using streams - ` example uses high-level streams - created by the :func:`open_connection` function in a coroutine. + * Another similar :ref:`example ` + using the high-level :func:`asyncio.open_connection` function + and streams. Set signal handlers for SIGINT and SIGTERM ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Register handlers for signals :py:data:`SIGINT` and :py:data:`SIGTERM` using -the :meth:`AbstractEventLoop.add_signal_handler` method:: +(This example only works on UNIX.) + +Register handlers for signals :py:data:`SIGINT` and :py:data:`SIGTERM` +using the :meth:`loop.add_signal_handler` method:: import asyncio import functools @@ -1286,16 +1533,17 @@ the :meth:`AbstractEventLoop.add_signal_handler` method:: print("got signal %s: exit" % signame) loop.stop() - loop = asyncio.get_event_loop() - for signame in ('SIGINT', 'SIGTERM'): - loop.add_signal_handler(getattr(signal, signame), - functools.partial(ask_exit, signame)) + async def main(): + loop = asyncio.get_running_loop() - print("Event loop running forever, press Ctrl+C to interrupt.") - print("pid %s: send SIGINT or SIGTERM to exit." % os.getpid()) - try: - loop.run_forever() - finally: - loop.close() + for signame in {'SIGINT', 'SIGTERM'}: + loop.add_signal_handler( + getattr(signal, signame), + functools.partial(ask_exit, signame)) + + await asyncio.sleep(3600) + + print("Event loop running for 1 hour, press Ctrl+C to interrupt.") + print(f"pid {os.getpid()}: send SIGINT or SIGTERM to exit.") -This example only works on UNIX. + asyncio.run(main()) diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst deleted file mode 100644 index 7f6e9535a8a5..000000000000 --- a/Doc/library/asyncio-eventloops.rst +++ /dev/null @@ -1,244 +0,0 @@ -.. currentmodule:: asyncio - -Event loops -=========== - -**Source code:** :source:`Lib/asyncio/events.py` - -Event loop functions --------------------- - -The following functions are convenient shortcuts to accessing the methods of the -global policy. Note that this provides access to the default policy, unless an -alternative policy was set by calling :func:`set_event_loop_policy` earlier in -the execution of the process. - -.. function:: get_event_loop() - - Equivalent to calling ``get_event_loop_policy().get_event_loop()``. - -.. function:: set_event_loop(loop) - - Equivalent to calling ``get_event_loop_policy().set_event_loop(loop)``. - -.. function:: new_event_loop() - - Equivalent to calling ``get_event_loop_policy().new_event_loop()``. - -.. function:: get_running_loop() - - Return the running event loop in the current OS thread. If there - is no running event loop a :exc:`RuntimeError` is raised. - - .. versionadded:: 3.7 - - -.. _asyncio-event-loops: - -Available event loops ---------------------- - -asyncio currently provides two implementations of event loops: -:class:`SelectorEventLoop` and :class:`ProactorEventLoop`. - -.. class:: SelectorEventLoop - - Event loop based on the :mod:`selectors` module. Subclass of - :class:`AbstractEventLoop`. - - Use the most efficient selector available on the platform. - - On Windows, only sockets are supported (ex: pipes are not supported): - see the `MSDN documentation of select - `_. - -.. class:: ProactorEventLoop - - Proactor event loop for Windows using "I/O Completion Ports" aka IOCP. - Subclass of :class:`AbstractEventLoop`. - - Availability: Windows. - - .. seealso:: - - `MSDN documentation on I/O Completion Ports - `_. - -Example to use a :class:`ProactorEventLoop` on Windows:: - - import asyncio, sys - - if sys.platform == 'win32': - loop = asyncio.ProactorEventLoop() - asyncio.set_event_loop(loop) - -.. _asyncio-platform-support: - -Platform support ----------------- - -The :mod:`asyncio` module has been designed to be portable, but each platform -still has subtle differences and may not support all :mod:`asyncio` features. - -Windows -^^^^^^^ - -Common limits of Windows event loops: - -- :meth:`~AbstractEventLoop.create_unix_connection` and - :meth:`~AbstractEventLoop.create_unix_server` are not supported: the socket - family :data:`socket.AF_UNIX` is specific to UNIX -- :meth:`~AbstractEventLoop.add_signal_handler` and - :meth:`~AbstractEventLoop.remove_signal_handler` are not supported -- :meth:`EventLoopPolicy.set_child_watcher` is not supported. - :class:`ProactorEventLoop` supports subprocesses. It has only one - implementation to watch child processes, there is no need to configure it. - -:class:`SelectorEventLoop` specific limits: - -- :class:`~selectors.SelectSelector` is used which only supports sockets - and is limited to 512 sockets. -- :meth:`~AbstractEventLoop.add_reader` and :meth:`~AbstractEventLoop.add_writer` only - accept file descriptors of sockets -- Pipes are not supported - (ex: :meth:`~AbstractEventLoop.connect_read_pipe`, - :meth:`~AbstractEventLoop.connect_write_pipe`) -- :ref:`Subprocesses ` are not supported - (ex: :meth:`~AbstractEventLoop.subprocess_exec`, - :meth:`~AbstractEventLoop.subprocess_shell`) - -:class:`ProactorEventLoop` specific limits: - -- :meth:`~AbstractEventLoop.create_datagram_endpoint` (UDP) is not supported -- :meth:`~AbstractEventLoop.add_reader` and :meth:`~AbstractEventLoop.add_writer` are - not supported - -The resolution of the monotonic clock on Windows is usually around 15.6 msec. -The best resolution is 0.5 msec. The resolution depends on the hardware -(availability of `HPET -`_) and on the Windows -configuration. See :ref:`asyncio delayed calls `. - -.. versionchanged:: 3.5 - - :class:`ProactorEventLoop` now supports SSL. - - -Mac OS X -^^^^^^^^ - -Character devices like PTY are only well supported since Mavericks (Mac OS -10.9). They are not supported at all on Mac OS 10.5 and older. - -On Mac OS 10.6, 10.7 and 10.8, the default event loop is -:class:`SelectorEventLoop` which uses :class:`selectors.KqueueSelector`. -:class:`selectors.KqueueSelector` does not support character devices on these -versions. The :class:`SelectorEventLoop` can be used with -:class:`~selectors.SelectSelector` or :class:`~selectors.PollSelector` to -support character devices on these versions of Mac OS X. Example:: - - import asyncio - import selectors - - selector = selectors.SelectSelector() - loop = asyncio.SelectorEventLoop(selector) - asyncio.set_event_loop(loop) - - -Event loop policies and the default policy ------------------------------------------- - -Event loop management is abstracted with a *policy* pattern, to provide maximal -flexibility for custom platforms and frameworks. Throughout the execution of a -process, a single global policy object manages the event loops available to the -process based on the calling context. A policy is an object implementing the -:class:`AbstractEventLoopPolicy` interface. - -For most users of :mod:`asyncio`, policies never have to be dealt with -explicitly, since the default global policy is sufficient (see below). - -The module-level functions -:func:`get_event_loop` and :func:`set_event_loop` provide convenient access to -event loops managed by the default policy. - - -Event loop policy interface ---------------------------- - -An event loop policy must implement the following interface: - -.. class:: AbstractEventLoopPolicy - - Event loop policy. - - .. method:: get_event_loop() - - Get the event loop for the current context. - - Returns an event loop object implementing the :class:`AbstractEventLoop` - interface. In case called from coroutine, it returns the currently - running event loop. - - Raises an exception in case no event loop has been set for the current - context and the current policy does not specify to create one. It must - never return ``None``. - - .. versionchanged:: 3.6 - - .. method:: set_event_loop(loop) - - Set the event loop for the current context to *loop*. - - .. method:: new_event_loop() - - Create and return a new event loop object according to this policy's - rules. - - If there's need to set this loop as the event loop for the current - context, :meth:`set_event_loop` must be called explicitly. - - -The default policy defines context as the current thread, and manages an event -loop per thread that interacts with :mod:`asyncio`. An exception to this rule -happens when :meth:`~AbstractEventLoopPolicy.get_event_loop` is called from a -running future/coroutine, in which case it will return the current loop -running that future/coroutine. - -If the current thread doesn't already have an event loop associated with it, -the default policy's :meth:`~AbstractEventLoopPolicy.get_event_loop` method -creates one when called from the main thread, but raises :exc:`RuntimeError` -otherwise. - - -Access to the global loop policy --------------------------------- - -.. function:: get_event_loop_policy() - - Get the current event loop policy. - -.. function:: set_event_loop_policy(policy) - - Set the current event loop policy. If *policy* is ``None``, the default - policy is restored. - - -Customizing the event loop policy ---------------------------------- - -To implement a new event loop policy, it is recommended you subclass the -concrete default event loop policy :class:`DefaultEventLoopPolicy` -and override the methods for which you want to change behavior, for example:: - - class MyEventLoopPolicy(asyncio.DefaultEventLoopPolicy): - - def get_event_loop(self): - """Get the event loop. - - This may be None or an instance of EventLoop. - """ - loop = super().get_event_loop() - # Do something with loop ... - return loop - - asyncio.set_event_loop_policy(MyEventLoopPolicy()) diff --git a/Doc/library/asyncio-exceptions.rst b/Doc/library/asyncio-exceptions.rst new file mode 100644 index 000000000000..bcd3599477f7 --- /dev/null +++ b/Doc/library/asyncio-exceptions.rst @@ -0,0 +1,88 @@ +.. currentmodule:: asyncio + + +========== +Exceptions +========== + + +.. exception:: TimeoutError + + The operation has exceeded the given deadline. + + .. note:: + This exception is different from the builtin :exc:`TimeoutError` + exception. + + +.. exception:: CancelledError + + The operation has been cancelled. + + This exception can be caught to perform custom operations on + when asyncio Tasks are cancelled. In almost all situations the + exception must always be re-raised. + + .. note:: + This exception is a subclass of :exc:`Exception`, so it can be + accidentally suppressed by ``try..except`` block:: + + try: + await operation + except Exception: + # The cancellation is broken because the *except* block + # suppresses the CancelledError exception. + log.log('an error has occurred') + + Instead, the following pattern should be used:: + + try: + await operation + except asyncio.CancelledError: + raise + except Exception: + log.log('an error has occurred') + + +.. exception:: InvalidStateError + + Invalid internal state of :class:`Task` or :class:`Future`. + + Can be raised in situations like setting a result value for a + *Future* object that already has a result value set. + + +.. exception:: SendfileNotAvailableError + + The "sendfile" syscall for is not available for the given + socket or file type. + + A subclass of :exc:`RuntimeError`. + + +.. exception:: IncompleteReadError + + Incomplete read error. + + Raised by :ref:`asyncio streams ` APIs. + + This exception is a subclass of :exc:`EOFError`. + + .. attribute:: expected + + Total number (:class:`int`) of expected bytes. + + .. attribute:: partial + + Read :class:`bytes` string before the end of stream was reached. + + +.. exception:: LimitOverrunError + + Reached the buffer limit while looking for a separator. + + Raised by :ref:`asyncio streams ` APIs. + + .. attribute:: consumed + + Total number of to be consumed bytes. diff --git a/Doc/library/asyncio-platforms.rst b/Doc/library/asyncio-platforms.rst new file mode 100644 index 000000000000..afdbce67cdd4 --- /dev/null +++ b/Doc/library/asyncio-platforms.rst @@ -0,0 +1,105 @@ +.. currentmodule:: asyncio + + +.. _asyncio-platform-support: + + +================= +Platforms Support +================= + +The :mod:`asyncio` module has been designed to be portable, +but some platforms have subtle differences and limitations. + + +All Platforms +============= + +* :meth:`loop.add_reader` and :meth:`loop.add_writer` + cannot be used to monitor file IO. + + +Windows +======= + +All event loops on Windows do not support the following methods: + +* :meth:`loop.create_unix_connection` and + :meth:`loop.create_unix_server` are not supported. + The :data:`socket.AF_UNIX` socket family is specific to UNIX/ + +* :meth:`loop.add_signal_handler` and + :meth:`loop.remove_signal_handler` are not supported. + +:class:`SelectorEventLoop` has the following limitations: + +* :class:`~selectors.SelectSelector` is used to wait on socket events: + it supports sockets and is limited to 512 sockets. + +* :meth:`loop.add_reader` and :meth:`loop.add_writer` only accept + socket handles (e.g. pipe file descriptors are not supported). + +* Pipes are not supported, so the :meth:`loop.connect_read_pipe` + and :meth:`loop.connect_write_pipe` methods are not implemented. + +* :ref:`Subprocesses ` are not supported, i.e. + :meth:`loop.subprocess_exec` and :meth:`loop.subprocess_shell` + methods are not implemented. + +:class:`ProactorEventLoop` has the following limitations: + +* The :meth:`loop.create_datagram_endpoint` method + is not supported. + +* The :meth:`loop.add_reader` and :meth:`loop.add_writer` + methods are not supported. + +The resolution of the monotonic clock on Windows is usually around 15.6 +msec. The best resolution is 0.5 msec. The resolution depends on the +hardware (availability of `HPET +`_) and on the +Windows configuration. + + +.. _asyncio-windows-subprocess: + +Subprocess Support on Windows +----------------------------- + +:class:`SelectorEventLoop` on Windows does not support subproceses, +so :class:`ProactorEventLoop` should be used instead:: + + import asyncio + + asyncio.set_event_loop_policy( + asyncio.WindowsProactorEventLoopPolicy()) + + asyncio.run(your_code()) + + +The :meth:`policy.set_child_watcher() +` function is also +not supported, as :class:`ProactorEventLoop` has a different mechanism +to watch child processes. + + +macOS +===== + +Modern macOS versions are fully supported. + +.. rubric:: macOS <= 10.8 + +On macOS 10.6, 10.7 and 10.8, the default event loop +uses :class:`selectors.KqueueSelector`, which does not support +character devices on these versions. The :class:`SelectorEventLoop` +can be manually configured to use :class:`~selectors.SelectSelector` +or :class:`~selectors.PollSelector` to support character devices on +these older versions of macOS. Example:: + + import asyncio + import selectors + + selector = selectors.SelectSelector() + loop = asyncio.SelectorEventLoop(selector) + asyncio.set_event_loop(loop) diff --git a/Doc/library/asyncio-policy.rst b/Doc/library/asyncio-policy.rst new file mode 100644 index 000000000000..7bb1c1587b7a --- /dev/null +++ b/Doc/library/asyncio-policy.rst @@ -0,0 +1,222 @@ +.. currentmodule:: asyncio + + +.. _asyncio-policies: + +======== +Policies +======== + +An event loop policy, a global per-process object, controls +management of the event loop. Each event loop has a default +policy, which can be changed and customized using the API. + +A policy defines the notion of context and manages a +separate event loop per context. The default policy +defines context to be the current thread. + +By using a custom event loop policy, the behavior of +:func:`get_event_loop`, :func:`set_event_loop`, and +:func:`new_event_loop` functions can be customized. + +Policy objects should implement the APIs defined +in the abstract base class :class:`AbstractEventLoopPolicy`. + + +Access the Policy +================= + +The following functions can be used to get and set the policy +for the current process: + +.. function:: get_event_loop_policy() + + Return the current process-wide policy. + +.. function:: set_event_loop_policy(policy) + + Set the current process-wide policy to *policy*. + + If *policy* is set to ``None``, the default policy is restored. + + +Policy Objects +============== + +The abstract event loop policy base class is defined as follows: + +.. class:: AbstractEventLoopPolicy + + An abstract base class for asyncio policies. + + .. method:: get_event_loop() + + Get the event loop for the current context. + + Return an event loop object implementing the + :class:`AbstractEventLoop` interface. + + This method should never return ``None``. + + .. versionchanged:: 3.6 + + .. method:: set_event_loop(loop) + + Set the event loop for the current context to *loop*. + + .. method:: new_event_loop() + + Create and return a new event loop object. + + This method should never return ``None``. + + .. method:: get_child_watcher() + + Get a child process watcher object. + + Return a watcher object implementing the + :class:`AbstractChildWatcher` interface. + + This function is Unix specific. + + .. method:: set_child_watcher(watcher) + + Get the current child process watcher to *watcher*. + + This function is Unix specific. + + +asyncio ships with the following built-in policies: + + +.. class:: DefaultEventLoopPolicy + + The default asyncio policy. Uses :class:`SelectorEventLoop` + on both Unix and Windows platforms. + + There is no need to install the default policy manually; asyncio + is configured to use it automatically. + + +.. class:: WindowsProactorEventLoopPolicy + + An alternative event loop policy that uses the + :class:`ProactorEventLoop` event loop implementation. + + Availability: Windows. + + +Process Watchers +================ + +A process watcher allows customization of how an event loop monitors +child processes on Unix. Specifically, the event loop needs to know +when a child process has finished its execution. + +In asyncio, child processes are created with +:func:`create_subprocess_exec` and :meth:`loop.subprocess_exec` +functions. + +asyncio defines an abstract base class :class:`AbstractChildWatcher` +that child watchers should implement, and has two different +implementations: :class:`SafeChildWatcher` (configured to be used +by default) and :class:`FastChildWatcher`. + +See also the :ref:`Subprocess and Threads ` +section. + +The following two functions can be used to customize the watcher +implementation used by the asyncio event loop: + +.. function:: get_child_watcher() + + Return the current child watcher for the current policy. + +.. function:: set_child_watcher(watcher) + + Set the current child watcher to *watcher* for the current + policy. *watcher* must implement methods defined in the + :class:`AbstractChildWatcher` base class. + +.. note:: + Third-party event loops implementations might not support + custom child watchers. For such event loops, using + :func:`set_child_watcher` might have no effect or even can + be prohibited. + +.. class:: AbstractChildWatcher + + .. method:: add_child_handler(pid, callback, \*args) + + Register a new child handler. + + Arrange for ``callback(pid, returncode, *args)`` to be called + when a process with PID equal to *pid* terminates. Specifying + another callback for the same process replaces the previous + handler. + + *callback* callable must be thread-safe. + + .. method:: remove_child_handler(pid) + + Removes the handler for process with PID equal to *pid*. + + The function returns ``True`` if the handler was successfully + removed, ``False`` if there was nothing to remove. + + .. method:: attach_loop(loop) + + Attach the watcher to an event loop. + + If the watcher was previously attached to an event loop, then + it is first detached before attaching to the new loop. + + Note: loop may be ``None``. + + .. method:: close() + + Close the watcher. + + This method has to be called to ensure that underlying + resources are cleaned-up. + +.. class:: SafeChildWatcher + + This implementation avoids disrupting other code spawning processes + by polling every process explicitly on a :py:data:`SIGCHLD` signal. + + This is a safe solution but it has a significant overhead when + handling a big number of processes (*O(n)* each time a + :py:data:`SIGCHLD` is received). + + asyncio uses this implementation by default. + +.. class:: FastChildWatcher + + This implementation reaps every terminated processes by calling + ``os.waitpid(-1)`` directly, possibly breaking other code spawning + processes and waiting for their termination. + + There is no noticeable overhead when handling a big number of + children (*O(1)* each time a child terminates). + + +Custom Policies +=============== + +To implement a new event loop policy, it is recommended to subclass +:class:`DefaultEventLoopPolicy` and override the methods for which +custom behavior is wanted, e.g.:: + + class MyEventLoopPolicy(asyncio.DefaultEventLoopPolicy): + + def get_event_loop(self): + """Get the event loop. + + This may be None or an instance of EventLoop. + """ + loop = super().get_event_loop() + # Do something with loop ... + return loop + + asyncio.set_event_loop_policy(MyEventLoopPolicy()) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index 9a08a4a49021..348ced57772f 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -1,12 +1,65 @@ .. currentmodule:: asyncio -+++++++++++++++++++++++++++++++++++++++++++++ -Transports and protocols (callback based API) -+++++++++++++++++++++++++++++++++++++++++++++ -**Source code:** :source:`Lib/asyncio/transports.py` +======================== +Transports and Protocols +======================== + +.. rubric:: Preface + +Transports and Protocols are used by **low-level** event loop +APIs such as :meth:`loop.create_connection`. They require using +callback-based programming style and enable high-performance +implementations of network or IPC protocols (e.g. HTTP). + +Essentially, transports and protocols should only be used in +libraries and frameworks and never in high-level asyncio +applications. + +This documentation page covers both `Transports`_ and `Protocols`_. + +.. rubric:: Introduction + +At the highest level, the transport is concerned with *how* bytes +are transmitted, while the protocol determines *which* bytes to +transmit (and to some extent when). + +A different way of saying the same thing: a transport is an +abstraction for a socket (or similar I/O endpoint) while a protocol +is an abstraction for an application, from the transport's point +of view. + +Yet another view is simply that the transport and protocol interfaces +together define an abstract interface for using network I/O and +interprocess I/O. + +There is always a 1:1 relationship between transport and protocol +objects: the protocol calls transport methods to send data, +while the transport calls protocol methods to pass it data that +has been received. + +Most of connection oriented event loop methods +(such as :meth:`loop.create_connection`) usually accept a +*protocol_factory* argument used to create a *Protocol* object +for an accepted connection, represented by a *Transport* object. +Such methods usually return a tuple of ``(transport, protocol)``. + +.. rubric:: Contents + +This documentation page contains the following sections: + +* The `Transports`_ section documents asyncio :class:`BaseTransport`, + :class:`ReadTransport`, :class:`WriteTransport`, :class:`Transport`, + :class:`DatagramTransport`, and :class:`SubprocessTransport` + classes. + +* The `Protocols`_ section documents asyncio :class:`BaseProtocol`, + :class:`Protocol`, :class:`BufferedProtocol`, + :class:`DatagramProtocol`, and :class:`SubprocessProtocol` classes. + +* The `Examples`_ section showcases how to work with transports, + protocols, and low-level event loop APIs. -**Source code:** :source:`Lib/asyncio/protocols.py` .. _asyncio-transport: @@ -14,293 +67,356 @@ Transports ========== Transports are classes provided by :mod:`asyncio` in order to abstract -various kinds of communication channels. You generally won't instantiate -a transport yourself; instead, you will call an :class:`AbstractEventLoop` method -which will create the transport and try to initiate the underlying -communication channel, calling you back when it succeeds. +various kinds of communication channels. -Once the communication channel is established, a transport is always -paired with a :ref:`protocol ` instance. The protocol can -then call the transport's methods for various purposes. +Transport objects are always instantiated by an +ref:`asyncio event loop `. -:mod:`asyncio` currently implements transports for TCP, UDP, SSL, and -subprocess pipes. The methods available on a transport depend on -the transport's kind. +asyncio implements transports for TCP, UDP, SSL, and subprocess pipes. +The methods available on a transport depend on the transport's kind. The transport classes are :ref:`not thread safe `. -.. versionchanged:: 3.6 - The socket option ``TCP_NODELAY`` is now set by default. - -BaseTransport -------------- +Transports Hierarchy +-------------------- .. class:: BaseTransport - Base class for transports. + Base class for all transports. Contains methods that all + asyncio transports share. - .. method:: close() +.. class:: WriteTransport(BaseTransport) - Close the transport. If the transport has a buffer for outgoing - data, buffered data will be flushed asynchronously. No more data - will be received. After all buffered data is flushed, the - protocol's :meth:`connection_lost` method will be called with - :const:`None` as its argument. + A base transport for write-only connections. - .. method:: is_closing() + Instances of the *WriteTransport* class are returned from + the :meth:`loop.connect_write_pipe` event loop method and + are also used by subprocess-related methods like + :meth:`loop.subprocess_exec`. - Return ``True`` if the transport is closing or is closed. +.. class:: ReadTransport(BaseTransport) - .. versionadded:: 3.5.1 + A base transport for read-only connections. - .. method:: get_extra_info(name, default=None) + Instances of the *ReadTransport* class are returned from + the :meth:`loop.connect_read_pipe` event loop method and + are also used by subprocess-related methods like + :meth:`loop.subprocess_exec`. - Return optional transport information. *name* is a string representing - the piece of transport-specific information to get, *default* is the - value to return if the information doesn't exist. +.. class:: Transport(WriteTransport, ReadTransport) - This method allows transport implementations to easily expose - channel-specific information. + Interface representing a bidirectional transport, such as a + TCP connection. - * socket: + The user never instantiates a transport directly; they call a + utility function, passing it a protocol factory and other + information necessary to create the transport and protocol. - - ``'peername'``: the remote address to which the socket is connected, - result of :meth:`socket.socket.getpeername` (``None`` on error) - - ``'socket'``: :class:`socket.socket` instance - - ``'sockname'``: the socket's own address, - result of :meth:`socket.socket.getsockname` + Instances of the *Transport* class are returned from or used by + event loop methods like :meth:`loop.create_connection`, + :meth:`loop.create_unix_connection`, + :meth:`loop.create_server`, :meth:`loop.sendfile`, etc. - * SSL socket: - - ``'compression'``: the compression algorithm being used as a string, - or ``None`` if the connection isn't compressed; result of - :meth:`ssl.SSLSocket.compression` - - ``'cipher'``: a three-value tuple containing the name of the cipher - being used, the version of the SSL protocol that defines its use, and - the number of secret bits being used; result of - :meth:`ssl.SSLSocket.cipher` - - ``'peercert'``: peer certificate; result of - :meth:`ssl.SSLSocket.getpeercert` - - ``'sslcontext'``: :class:`ssl.SSLContext` instance - - ``'ssl_object'``: :class:`ssl.SSLObject` or :class:`ssl.SSLSocket` - instance +.. class:: DatagramTransport(BaseTransport) - * pipe: + A transport for datagram (UDP) connections. - - ``'pipe'``: pipe object + Instances of the *DatagramTransport* class are returned from + the :meth:`loop.create_datagram_endpoint` event loop method. - * subprocess: - - ``'subprocess'``: :class:`subprocess.Popen` instance +.. class:: SubprocessTransport(BaseTransport) - .. method:: set_protocol(protocol) + An abstraction to represent a connection between a parent and its + child OS process. - Set a new protocol. Switching protocol should only be done when both - protocols are documented to support the switch. + Instances of the *SubprocessTransport* class are returned from + event loop methods :meth:`loop.subprocess_shell` and + :meth:`loop.subprocess_exec`. - .. versionadded:: 3.5.3 - .. method:: get_protocol +Base Transport +-------------- - Return the current protocol. +.. method:: BaseTransport.close() - .. versionadded:: 3.5.3 + Close the transport. - .. versionchanged:: 3.5.1 - ``'ssl_object'`` info was added to SSL sockets. + If the transport has a buffer for outgoing + data, buffered data will be flushed asynchronously. No more data + will be received. After all buffered data is flushed, the + protocol's :meth:`protocol.connection_lost() + ` method will be called with + :const:`None` as its argument. +.. method:: BaseTransport.is_closing() -ReadTransport -------------- + Return ``True`` if the transport is closing or is closed. -.. class:: ReadTransport +.. method:: BaseTransport.get_extra_info(name, default=None) - Interface for read-only transports. + Return information about the transport or underlying resources + it uses. - .. method:: is_reading() + *name* is a string representing the piece of transport-specific + information to get. - Return ``True`` if the transport is receiving new data. + *default* is the value to return if the information is not + available, or if the transport does not support querying it + with the given third-party event loop implementation or on the + current platform. - .. versionadded:: 3.7 + For example, the following code attempts to get the underlying + socket object of the transport:: - .. method:: pause_reading() + sock = transport.get_extra_info('socket') + if sock is not None: + print(sock.getsockopt(...)) - Pause the receiving end of the transport. No data will be passed to - the protocol's :meth:`data_received` method until :meth:`resume_reading` - is called. + Categories of information that can be queried on some transports: - .. versionchanged:: 3.7 - The method is idempotent, i.e. it can be called when the - transport is already paused or closed. + * socket: - .. method:: resume_reading() + - ``'peername'``: the remote address to which the socket is + connected, result of :meth:`socket.socket.getpeername` + (``None`` on error) - Resume the receiving end. The protocol's :meth:`data_received` method - will be called once again if some data is available for reading. + - ``'socket'``: :class:`socket.socket` instance - .. versionchanged:: 3.7 - The method is idempotent, i.e. it can be called when the - transport is already reading. + - ``'sockname'``: the socket's own address, + result of :meth:`socket.socket.getsockname` + * SSL socket: -WriteTransport --------------- + - ``'compression'``: the compression algorithm being used as a + string, or ``None`` if the connection isn't compressed; result + of :meth:`ssl.SSLSocket.compression` -.. class:: WriteTransport + - ``'cipher'``: a three-value tuple containing the name of the + cipher being used, the version of the SSL protocol that defines + its use, and the number of secret bits being used; result of + :meth:`ssl.SSLSocket.cipher` - Interface for write-only transports. + - ``'peercert'``: peer certificate; result of + :meth:`ssl.SSLSocket.getpeercert` - .. method:: abort() + - ``'sslcontext'``: :class:`ssl.SSLContext` instance - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. + - ``'ssl_object'``: :class:`ssl.SSLObject` or + :class:`ssl.SSLSocket` instance - .. method:: can_write_eof() + * pipe: - Return :const:`True` if the transport supports :meth:`write_eof`, - :const:`False` if not. + - ``'pipe'``: pipe object - .. method:: get_write_buffer_size() + * subprocess: - Return the current size of the output buffer used by the transport. + - ``'subprocess'``: :class:`subprocess.Popen` instance - .. method:: get_write_buffer_limits() +.. method:: BaseTransport.set_protocol(protocol) - Get the *high*- and *low*-water limits for write flow control. Return a - tuple ``(low, high)`` where *low* and *high* are positive number of - bytes. + Set a new protocol. - Use :meth:`set_write_buffer_limits` to set the limits. + Switching protocol should only be done when both + protocols are documented to support the switch. - .. versionadded:: 3.4.2 +.. method:: BaseTransport.get_protocol() - .. method:: set_write_buffer_limits(high=None, low=None) + Return the current protocol. - Set the *high*- and *low*-water limits for write flow control. - These two values (measured in number of - bytes) control when the protocol's - :meth:`pause_writing` and :meth:`resume_writing` methods are called. - If specified, the low-water limit must be less than or equal to the - high-water limit. Neither *high* nor *low* can be negative. +Read-only Transports +-------------------- - :meth:`pause_writing` is called when the buffer size becomes greater - than or equal to the *high* value. If writing has been paused, - :meth:`resume_writing` is called when the buffer size becomes less - than or equal to the *low* value. +.. method:: ReadTransport.is_reading() - The defaults are implementation-specific. If only the - high-water limit is given, the low-water limit defaults to an - implementation-specific value less than or equal to the - high-water limit. Setting *high* to zero forces *low* to zero as - well, and causes :meth:`pause_writing` to be called whenever the - buffer becomes non-empty. Setting *low* to zero causes - :meth:`resume_writing` to be called only once the buffer is empty. - Use of zero for either limit is generally sub-optimal as it - reduces opportunities for doing I/O and computation - concurrently. + Return ``True`` if the transport is receiving new data. - Use :meth:`get_write_buffer_limits` to get the limits. + .. versionadded:: 3.7 - .. method:: write(data) +.. method:: ReadTransport.pause_reading() - Write some *data* bytes to the transport. + Pause the receiving end of the transport. No data will be passed to + the protocol's :meth:`protocol.data_received() ` + method until :meth:`resume_reading` is called. - This method does not block; it buffers the data and arranges for it - to be sent out asynchronously. + .. versionchanged:: 3.7 + The method is idempotent, i.e. it can be called when the + transport is already paused or closed. - .. method:: writelines(list_of_data) +.. method:: ReadTransport.resume_reading() - Write a list (or any iterable) of data bytes to the transport. - This is functionally equivalent to calling :meth:`write` on each - element yielded by the iterable, but may be implemented more efficiently. + Resume the receiving end. The protocol's + :meth:`protocol.data_received() ` method + will be called once again if some data is available for reading. - .. method:: write_eof() + .. versionchanged:: 3.7 + The method is idempotent, i.e. it can be called when the + transport is already reading. - Close the write end of the transport after flushing buffered data. - Data may still be received. - This method can raise :exc:`NotImplementedError` if the transport - (e.g. SSL) doesn't support half-closes. +Write-only Transports +--------------------- +.. method:: WriteTransport.abort() -DatagramTransport ------------------ + Close the transport immediately, without waiting for pending operations + to complete. Buffered data will be lost. No more data will be received. + The protocol's :meth:`protocol.connection_lost() + ` method will eventually be + called with :const:`None` as its argument. -.. method:: DatagramTransport.sendto(data, addr=None) +.. method:: WriteTransport.can_write_eof() - Send the *data* bytes to the remote peer given by *addr* (a - transport-dependent target address). If *addr* is :const:`None`, the - data is sent to the target address given on transport creation. + Return :const:`True` if the transport supports + :meth:`~WriteTransport.write_eof`, :const:`False` if not. + +.. method:: WriteTransport.get_write_buffer_size() + + Return the current size of the output buffer used by the transport. + +.. method:: WriteTransport.get_write_buffer_limits() + + Get the *high*- and *low*-water limits for write flow control. Return a + tuple ``(low, high)`` where *low* and *high* are positive number of + bytes. + + Use :meth:`set_write_buffer_limits` to set the limits. + + .. versionadded:: 3.4.2 + +.. method:: WriteTransport.set_write_buffer_limits(high=None, low=None) + + Set the *high*- and *low*-water limits for write flow control. + + These two values (measured in number of + bytes) control when the protocol's + :meth:`protocol.pause_writing() ` + and :meth:`protocol.resume_writing() ` + methods are called. If specified, the low-water limit must be less + than or equal to the high-water limit. Neither *high* nor *low* + can be negative. + + :meth:`~BaseProtocol.pause_writing` is called when the buffer size + becomes greater than or equal to the *high* value. If writing has + been paused, :meth:`~BaseProtocol.resume_writing` is called when + the buffer size becomes less than or equal to the *low* value. + + The defaults are implementation-specific. If only the + high-water limit is given, the low-water limit defaults to an + implementation-specific value less than or equal to the + high-water limit. Setting *high* to zero forces *low* to zero as + well, and causes :meth:`~BaseProtocol.pause_writing` to be called + whenever the buffer becomes non-empty. Setting *low* to zero causes + :meth:`~BaseProtocol.resume_writing` to be called only once the + buffer is empty. Use of zero for either limit is generally + sub-optimal as it reduces opportunities for doing I/O and + computation concurrently. + + Use :meth:`~WriteTransport.get_write_buffer_limits` + to get the limits. + +.. method:: WriteTransport.write(data) + + Write some *data* bytes to the transport. This method does not block; it buffers the data and arranges for it to be sent out asynchronously. +.. method:: WriteTransport.writelines(list_of_data) + + Write a list (or any iterable) of data bytes to the transport. + This is functionally equivalent to calling :meth:`write` on each + element yielded by the iterable, but may be implemented more + efficiently. + +.. method:: WriteTransport.write_eof() + + Close the write end of the transport after flushing buffered data. + Data may still be received. + + This method can raise :exc:`NotImplementedError` if the transport + (e.g. SSL) doesn't support half-closes. + + +Datagram Transports +------------------- + +.. method:: DatagramTransport.sendto(data, addr=None) + + Send the *data* bytes to the remote peer given by *addr* (a + transport-dependent target address). If *addr* is :const:`None`, + the data is sent to the target address given on transport + creation. + + This method does not block; it buffers the data and arranges + for it to be sent out asynchronously. + .. method:: DatagramTransport.abort() - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. + Close the transport immediately, without waiting for pending + operations to complete. Buffered data will be lost. + No more data will be received. The protocol's + :meth:`protocol.connection_lost() ` + method will eventually be called with :const:`None` as its argument. -BaseSubprocessTransport ------------------------ +.. _asyncio-subprocess-transports: -.. class:: BaseSubprocessTransport +Subprocess Transports +--------------------- - .. method:: get_pid() +.. method:: SubprocessTransport.get_pid() - Return the subprocess process id as an integer. + Return the subprocess process id as an integer. - .. method:: get_pipe_transport(fd) +.. method:: SubprocessTransport.get_pipe_transport(fd) - Return the transport for the communication pipe corresponding to the - integer file descriptor *fd*: + Return the transport for the communication pipe corresponding to the + integer file descriptor *fd*: - * ``0``: readable streaming transport of the standard input (*stdin*), - or :const:`None` if the subprocess was not created with ``stdin=PIPE`` - * ``1``: writable streaming transport of the standard output (*stdout*), - or :const:`None` if the subprocess was not created with ``stdout=PIPE`` - * ``2``: writable streaming transport of the standard error (*stderr*), - or :const:`None` if the subprocess was not created with ``stderr=PIPE`` - * other *fd*: :const:`None` + * ``0``: readable streaming transport of the standard input (*stdin*), + or :const:`None` if the subprocess was not created with ``stdin=PIPE`` + * ``1``: writable streaming transport of the standard output (*stdout*), + or :const:`None` if the subprocess was not created with ``stdout=PIPE`` + * ``2``: writable streaming transport of the standard error (*stderr*), + or :const:`None` if the subprocess was not created with ``stderr=PIPE`` + * other *fd*: :const:`None` - .. method:: get_returncode() +.. method:: SubprocessTransport.get_returncode() - Return the subprocess returncode as an integer or :const:`None` - if it hasn't returned, similarly to the - :attr:`subprocess.Popen.returncode` attribute. + Return the subprocess return code as an integer or :const:`None` + if it hasn't returned, similarly to the + :attr:`subprocess.Popen.returncode` attribute. - .. method:: kill() +.. method:: SubprocessTransport.kill() - Kill the subprocess, as in :meth:`subprocess.Popen.kill`. + Kill the subprocess, as in :meth:`subprocess.Popen.kill`. - On POSIX systems, the function sends SIGKILL to the subprocess. - On Windows, this method is an alias for :meth:`terminate`. + On POSIX systems, the function sends SIGKILL to the subprocess. + On Windows, this method is an alias for :meth:`terminate`. - .. method:: send_signal(signal) +.. method:: SubprocessTransport.send_signal(signal) - Send the *signal* number to the subprocess, as in - :meth:`subprocess.Popen.send_signal`. + Send the *signal* number to the subprocess, as in + :meth:`subprocess.Popen.send_signal`. - .. method:: terminate() +.. method:: SubprocessTransport.terminate() - Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. - This method is an alias for the :meth:`close` method. + Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. + This method is an alias for the :meth:`close` method. - On POSIX systems, this method sends SIGTERM to the subprocess. - On Windows, the Windows API function TerminateProcess() is called to - stop the subprocess. + On POSIX systems, this method sends SIGTERM to the subprocess. + On Windows, the Windows API function TerminateProcess() is called to + stop the subprocess. - .. method:: close() +.. method:: SubprocessTransport.close() - Ask the subprocess to stop by calling the :meth:`terminate` method if the - subprocess hasn't returned yet, and close transports of all pipes - (*stdin*, *stdout* and *stderr*). + Ask the subprocess to stop by calling the :meth:`terminate` method + if the subprocess hasn't returned yet, and close transports of all + pipes (*stdin*, *stdout* and *stderr*). .. _asyncio-protocol: @@ -308,65 +424,62 @@ BaseSubprocessTransport Protocols ========= -:mod:`asyncio` provides base classes that you can subclass to implement -your network protocols. Those classes are used in conjunction with -:ref:`transports ` (see below): the protocol parses incoming -data and asks for the writing of outgoing data, while the transport is -responsible for the actual I/O and buffering. +asyncio provides a set of abstract base classes that should be used +to implement network protocols. Those classes are meant to be used +together with :ref:`transports `. -When subclassing a protocol class, it is recommended you override certain -methods. Those methods are callbacks: they will be called by the transport -on certain events (for example when some data is received); you shouldn't -call them yourself, unless you are implementing a transport. +Subclasses of abstract base protocol classes can implement some or +all methods. All those methods are callbacks: they are called by +transports on certain events, for example when some data is received. +Base protocol methods are not supposed to be called by anything but +the corresponding transport. -.. note:: - All callbacks have default implementations, which are empty. Therefore, - you only need to implement the callbacks for the events in which you - are interested. +Base Protocols +-------------- + +.. class:: BaseProtocol -Protocol classes ----------------- + Base protocol with methods that all protocols share. -.. class:: Protocol +.. class:: Protocol(BaseProtocol) - The base class for implementing streaming protocols (for use with - e.g. TCP and SSL transports). + The base class for implementing streaming protocols + (TCP, Unix sockets, etc). -.. class:: BufferedProtocol +.. class:: BufferedProtocol(BaseProtocol) A base class for implementing streaming protocols with manual control of the receive buffer. - .. versionadded:: 3.7 - **Important:** this has been added to asyncio in Python 3.7 - *on a provisional basis*! Treat it as an experimental API that - might be changed or removed in Python 3.8. - -.. class:: DatagramProtocol +.. class:: DatagramProtocol(BaseProtocol) - The base class for implementing datagram protocols (for use with - e.g. UDP transports). + The base class for implementing datagram (UDP) protocols. -.. class:: SubprocessProtocol +.. class:: SubprocessProtocol(BaseProtocol) The base class for implementing protocols communicating with child - processes (through a set of unidirectional pipes). + processes (unidirectional pipes). -Connection callbacks --------------------- +Base Protocol +------------- + +All asyncio protocols can implement Base Protocol callbacks. -These callbacks may be called on :class:`Protocol`, :class:`DatagramProtocol` -and :class:`SubprocessProtocol` instances: +.. rubric:: Connection Callbacks + +Connection callbacks are called on all protocols, exactly once per +a successful connection. All other protocol callbacks can only be +called between those two methods. .. method:: BaseProtocol.connection_made(transport) Called when a connection is made. The *transport* argument is the transport representing the - connection. You are responsible for storing it somewhere - (e.g. as an attribute) if you need to. + connection. The protocol is responsible for storing the reference + to its transport. .. method:: BaseProtocol.connection_lost(exc) @@ -376,65 +489,77 @@ and :class:`SubprocessProtocol` instances: The latter means a regular EOF is received, or the connection was aborted or closed by this side of the connection. -:meth:`~BaseProtocol.connection_made` and :meth:`~BaseProtocol.connection_lost` -are called exactly once per successful connection. All other callbacks will be -called between those two methods, which allows for easier resource management -in your protocol implementation. -The following callbacks may be called only on :class:`SubprocessProtocol` -instances: +.. rubric:: Flow Control Callbacks -.. method:: SubprocessProtocol.pipe_data_received(fd, data) +Flow control callbacks can be called by transports to pause or +resume writing performed by the protocol. - Called when the child process writes data into its stdout or stderr pipe. - *fd* is the integer file descriptor of the pipe. *data* is a non-empty - bytes object containing the data. +See the documentation of the :meth:`~WriteTransport.set_write_buffer_limits` +method for more details. -.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) +.. method:: BaseProtocol.pause_writing() - Called when one of the pipes communicating with the child process - is closed. *fd* is the integer file descriptor that was closed. + Called when the transport's buffer goes over the high-water mark. -.. method:: SubprocessProtocol.process_exited() +.. method:: BaseProtocol.resume_writing() - Called when the child process has exited. + Called when the transport's buffer drains below the low-water mark. +If the buffer size equals the high-water mark, +:meth:`~BaseProtocol.pause_writing` is not called: the buffer size must +go strictly over. -Streaming protocols +Conversely, :meth:`~BaseProtocol.resume_writing` is called when the +buffer size is equal or lower than the low-water mark. These end +conditions are important to ensure that things go as expected when +either mark is zero. + + +Streaming Protocols ------------------- -The following callbacks are called on :class:`Protocol` instances: +Event methods, such as :meth:`loop.create_server`, +:meth:`loop.create_unix_server`, :meth:`loop.create_connection`, +:meth:`loop.create_unix_connection`, :meth:`loop.connect_accepted_socket`, +:meth:`loop.connect_read_pipe`, and :meth:`loop.connect_write_pipe` +accept factories that return streaming protocols. .. method:: Protocol.data_received(data) - Called when some data is received. *data* is a non-empty bytes object - containing the incoming data. + Called when some data is received. *data* is a non-empty bytes + object containing the incoming data. + + Whether the data is buffered, chunked or reassembled depends on + the transport. In general, you shouldn't rely on specific semantics + and instead make your parsing generic and flexible enough. However, + data is always received in the correct order. + + The method can be called an arbitrary number of times during + a connection. - .. note:: - Whether the data is buffered, chunked or reassembled depends on - the transport. In general, you shouldn't rely on specific semantics - and instead make your parsing generic and flexible enough. However, - data is always received in the correct order. + However, :meth:`protocol.eof_received() ` + is called at most once and, if called, + :meth:`protocol.data_received() ` + won't be called after it. .. method:: Protocol.eof_received() Called when the other end signals it won't send any more data - (for example by calling :meth:`write_eof`, if the other end also uses + (for example by calling :meth:`transport.write_eof() + `, if the other end also uses asyncio). This method may return a false value (including ``None``), in which case the transport will close itself. Conversely, if this method returns a true value, closing the transport is up to the protocol. Since the - default implementation returns ``None``, it implicitly closes the connection. + default implementation returns ``None``, it implicitly closes the + connection. - .. note:: - Some transports such as SSL don't support half-closed connections, - in which case returning true from this method will not prevent closing - the connection. + Some transports such as SSL don't support half-closed connections, + in which case returning true from this method will not prevent closing + the connection. -:meth:`data_received` can be called an arbitrary number of times during -a connection. However, :meth:`eof_received` is called at most once -and, if called, :meth:`data_received` won't be called after it. State machine: @@ -446,20 +571,18 @@ State machine: -> connection_lost -> end -Streaming protocols with manual receive buffer control ------------------------------------------------------- +Buffered Streaming Protocols +---------------------------- .. versionadded:: 3.7 - **Important:** :class:`BufferedProtocol` has been added to - asyncio in Python 3.7 *on a provisional basis*! Consider it as an - experimental API that might be changed or removed in Python 3.8. + **Important:** this has been added to asyncio in Python 3.7 + *on a provisional basis*! This is as an experimental API that + might be changed or removed completely in Python 3.8. +Buffered Protocols can be used with any event loop method +that supports `Streaming Protocols`_. -Event methods, such as :meth:`AbstractEventLoop.create_server` and -:meth:`AbstractEventLoop.create_connection`, accept factories that -return protocols that implement this interface. - -The idea of BufferedProtocol is that it allows to manually allocate +The idea of ``BufferedProtocol`` is that it allows to manually allocate and control the receive buffer. Event loops can then use the buffer provided by the protocol to avoid unnecessary data copies. This can result in noticeable performance improvement for protocols that @@ -489,13 +612,15 @@ instances: .. method:: BufferedProtocol.eof_received() - See the documentation of the :meth:`Protocol.eof_received` method. + See the documentation of the :meth:`protocol.eof_received() + ` method. -:meth:`get_buffer` can be called an arbitrary number of times during -a connection. However, :meth:`eof_received` is called at most once -and, if called, :meth:`get_buffer` and :meth:`buffer_updated` -won't be called after it. +:meth:`~BufferedProtocol.get_buffer` can be called an arbitrary number +of times during a connection. However, :meth:`protocol.eof_received() +` is called at most once +and, if called, :meth:`~BufferedProtocol.get_buffer` and +:meth:`~BufferedProtocol.buffer_updated` won't be called after it. State machine: @@ -509,10 +634,11 @@ State machine: -> connection_lost -> end -Datagram protocols +Datagram Protocols ------------------ -The following callbacks are called on :class:`DatagramProtocol` instances. +Datagram Protocol instances should be constructed by protocol +factories passed to the :meth:`loop.create_datagram_endpoint` method. .. method:: DatagramProtocol.datagram_received(data, addr) @@ -530,76 +656,116 @@ The following callbacks are called on :class:`DatagramProtocol` instances. In many conditions though, undeliverable datagrams will be silently dropped. +.. note:: -Flow control callbacks ----------------------- + On BSD systems (macOS, FreeBSD, etc.) flow control is not supported + for datagram protocols, because send failures caused by + writing too many packets cannot be detected easily. -These callbacks may be called on :class:`Protocol`, -:class:`DatagramProtocol` and :class:`SubprocessProtocol` instances: + The socket always appears 'ready' and excess packets are dropped; an + :class:`OSError` with ``errno`` set to :const:`errno.ENOBUFS` may + or may not be raised; if it is raised, it will be reported to + :meth:`DatagramProtocol.error_received` but otherwise ignored. -.. method:: BaseProtocol.pause_writing() - Called when the transport's buffer goes over the high-water mark. +.. _asyncio-subprocess-protocols: -.. method:: BaseProtocol.resume_writing() +Subprocess Protocols +-------------------- - Called when the transport's buffer drains below the low-water mark. +Datagram Protocol instances should be constructed by protocol +factories passed to the :meth:`loop.subprocess_exec` and +:meth:`loop.subprocess_shell` methods. +.. method:: SubprocessProtocol.pipe_data_received(fd, data) -:meth:`pause_writing` and :meth:`resume_writing` calls are paired -- -:meth:`pause_writing` is called once when the buffer goes strictly over -the high-water mark (even if subsequent writes increases the buffer size -even more), and eventually :meth:`resume_writing` is called once when the -buffer size reaches the low-water mark. + Called when the child process writes data into its stdout or stderr + pipe. -.. note:: - If the buffer size equals the high-water mark, - :meth:`pause_writing` is not called -- it must go strictly over. - Conversely, :meth:`resume_writing` is called when the buffer size is - equal or lower than the low-water mark. These end conditions - are important to ensure that things go as expected when either - mark is zero. + *fd* is the integer file descriptor of the pipe. -.. note:: - On BSD systems (OS X, FreeBSD, etc.) flow control is not supported - for :class:`DatagramProtocol`, because send failures caused by - writing too many packets cannot be detected easily. The socket - always appears 'ready' and excess packets are dropped; an - :class:`OSError` with errno set to :const:`errno.ENOBUFS` may or - may not be raised; if it is raised, it will be reported to - :meth:`DatagramProtocol.error_received` but otherwise ignored. + *data* is a non-empty bytes object containing the received data. + +.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) + + Called when one of the pipes communicating with the child process + is closed. + + *fd* is the integer file descriptor that was closed. + +.. method:: SubprocessProtocol.process_exited() + + Called when the child process has exited. -Coroutines and protocols ------------------------- +Examples +======== -Coroutines can be scheduled in a protocol method using :func:`ensure_future`, -but there is no guarantee made about the execution order. Protocols are not -aware of coroutines created in protocol methods and so will not wait for them. +.. _asyncio-tcp-echo-server-protocol: + +TCP Echo Server +--------------- + +TCP echo server using the :meth:`loop.create_server` method, send back +received data and close the connection:: + + import asyncio + + + class EchoServerClientProtocol(asyncio.Protocol): + def connection_made(self, transport): + peername = transport.get_extra_info('peername') + print('Connection from {}'.format(peername)) + self.transport = transport + + def data_received(self, data): + message = data.decode() + print('Data received: {!r}'.format(message)) + + print('Send: {!r}'.format(message)) + self.transport.write(data) + + print('Close the client socket') + self.transport.close() + + + async def main(): + # Get a reference to the event loop as we plan to use + # low-level APIs. + loop = asyncio.get_running_loop() + + server = await loop.create_server( + lambda: EchoServerClientProtocol(), + '127.0.0.1', 8888) -To have a reliable execution order, -use :ref:`stream objects ` in a -coroutine with ``await``. For example, the :meth:`StreamWriter.drain` -coroutine can be used to wait until the write buffer is flushed. + async with server: + await server.serve_forever() -Protocol examples -================= + asyncio.run(main()) + + +.. seealso:: + + The :ref:`TCP echo server using streams ` + example uses the high-level :func:`asyncio.start_server` function. .. _asyncio-tcp-echo-client-protocol: -TCP echo client protocol ------------------------- +TCP Echo Client +--------------- -TCP echo client using the :meth:`AbstractEventLoop.create_connection` method, send +TCP echo client using the :meth:`loop.create_connection` method, send data and wait until the connection is closed:: import asyncio + class EchoClientProtocol(asyncio.Protocol): - def __init__(self, message, loop): + def __init__(self, message, on_con_lost, loop): self.message = message self.loop = loop + self.on_con_lost = on_con_lost def connection_made(self, transport): transport.write(self.message.encode()) @@ -610,99 +776,99 @@ data and wait until the connection is closed:: def connection_lost(self, exc): print('The server closed the connection') - print('Stop the event loop') - self.loop.stop() - - loop = asyncio.get_event_loop() - message = 'Hello World!' - coro = loop.create_connection(lambda: EchoClientProtocol(message, loop), - '127.0.0.1', 8888) - loop.run_until_complete(coro) - loop.run_forever() - loop.close() - -The event loop is running twice. The -:meth:`~AbstractEventLoop.run_until_complete` method is preferred in this short -example to raise an exception if the server is not listening, instead of -having to write a short coroutine to handle the exception and stop the -running loop. At :meth:`~AbstractEventLoop.run_until_complete` exit, the loop is -no longer running, so there is no need to stop the loop in case of an error. + self.on_con_lost.set_result(True) + + + async def main(): + # Get a reference to the event loop as we plan to use + # low-level APIs. + loop = asyncio.get_running_loop() + + on_con_lost = loop.create_future() + message = 'Hello World!' + + transport, protocol = await loop.create_connection( + lambda: EchoClientProtocol(message, on_con_lost, loop), + '127.0.0.1', 8888) + + # Wait until the protocol signals that the connection + # is lost and close the transport. + try: + await on_con_lost + finally: + transport.close() + + + asyncio.run(main()) + .. seealso:: The :ref:`TCP echo client using streams ` - example uses the :func:`asyncio.open_connection` function. + example uses the high-level :func:`asyncio.open_connection` function. -.. _asyncio-tcp-echo-server-protocol: +.. _asyncio-udp-echo-server-protocol: -TCP echo server protocol ------------------------- +UDP Echo Server +--------------- -TCP echo server using the :meth:`AbstractEventLoop.create_server` method, send back -received data and close the connection:: +UDP echo server using the :meth:`loop.create_datagram_endpoint` +method, send back received data:: import asyncio - class EchoServerClientProtocol(asyncio.Protocol): + + class EchoServerProtocol: def connection_made(self, transport): - peername = transport.get_extra_info('peername') - print('Connection from {}'.format(peername)) self.transport = transport - def data_received(self, data): + def datagram_received(self, data, addr): message = data.decode() - print('Data received: {!r}'.format(message)) - - print('Send: {!r}'.format(message)) - self.transport.write(data) + print('Received %r from %s' % (message, addr)) + print('Send %r to %s' % (message, addr)) + self.transport.sendto(data, addr) - print('Close the client socket') - self.transport.close() - loop = asyncio.get_event_loop() - # Each client connection will create a new protocol instance - coro = loop.create_server(EchoServerClientProtocol, '127.0.0.1', 8888) - server = loop.run_until_complete(coro) + async def main(): + print("Starting UDP server") - # Serve requests until Ctrl+C is pressed - print('Serving on {}'.format(server.sockets[0].getsockname())) - try: - loop.run_forever() - except KeyboardInterrupt: - pass + # Get a reference to the event loop as we plan to use + # low-level APIs. + loop = asyncio.get_running_loop() - # Close the server - server.close() - loop.run_until_complete(server.wait_closed()) - loop.close() + # One protocol instance will be created to serve all + # client requests. + transport, protocol = await loop.create_datagram_endpoint( + lambda: EchoServerProtocol(), + local_addr=('127.0.0.1', 9999)) -:meth:`Transport.close` can be called immediately after -:meth:`WriteTransport.write` even if data are not sent yet on the socket: both -methods are asynchronous. ``await`` is not needed because these transport -methods are not coroutines. + try: + await asyncio.sleep(3600) # Serve for 1 hour. + finally: + transport.close() -.. seealso:: - The :ref:`TCP echo server using streams ` - example uses the :func:`asyncio.start_server` function. + asyncio.run(main()) .. _asyncio-udp-echo-client-protocol: -UDP echo client protocol ------------------------- +UDP Echo Client +--------------- -UDP echo client using the :meth:`AbstractEventLoop.create_datagram_endpoint` +UDP echo client using the :meth:`loop.create_datagram_endpoint` method, send data and close the transport when we received the answer:: import asyncio + class EchoClientProtocol: def __init__(self, message, loop): self.message = message self.loop = loop self.transport = None + self.on_con_lost = loop.create_future() def connection_made(self, transport): self.transport = transport @@ -719,75 +885,46 @@ method, send data and close the transport when we received the answer:: print('Error received:', exc) def connection_lost(self, exc): - print("Socket closed, stop the event loop") - loop = asyncio.get_event_loop() - loop.stop() + print("Connection closed") + self.on_con_lost.set_result(True) - loop = asyncio.get_event_loop() - message = "Hello World!" - connect = loop.create_datagram_endpoint( - lambda: EchoClientProtocol(message, loop), - remote_addr=('127.0.0.1', 9999)) - transport, protocol = loop.run_until_complete(connect) - loop.run_forever() - transport.close() - loop.close() + async def main(): + # Get a reference to the event loop as we plan to use + # low-level APIs. + loop = asyncio.get_running_loop() -.. _asyncio-udp-echo-server-protocol: - -UDP echo server protocol ------------------------- + message = "Hello World!" + transport, protocol = await loop.create_datagram_endpoint( + lambda: EchoClientProtocol(message, loop), + remote_addr=('127.0.0.1', 9999)) -UDP echo server using the :meth:`AbstractEventLoop.create_datagram_endpoint` -method, send back received data:: + try: + await protocol.on_con_lost + finally: + transport.close() - import asyncio - class EchoServerProtocol: - def connection_made(self, transport): - self.transport = transport - - def datagram_received(self, data, addr): - message = data.decode() - print('Received %r from %s' % (message, addr)) - print('Send %r to %s' % (message, addr)) - self.transport.sendto(data, addr) - - loop = asyncio.get_event_loop() - print("Starting UDP server") - # One protocol instance will be created to serve all client requests - listen = loop.create_datagram_endpoint( - EchoServerProtocol, local_addr=('127.0.0.1', 9999)) - transport, protocol = loop.run_until_complete(listen) - - try: - loop.run_forever() - except KeyboardInterrupt: - pass - - transport.close() - loop.close() + asyncio.run(main()) .. _asyncio-register-socket: -Register an open socket to wait for data using a protocol ---------------------------------------------------------- +Connecting Existing Sockets +--------------------------- Wait until a socket receives data using the -:meth:`AbstractEventLoop.create_connection` method with a protocol, and then close -the event loop :: +:meth:`loop.create_connection` method with a protocol:: import asyncio - from socket import socketpair + import socket - # Create a pair of connected sockets - rsock, wsock = socketpair() - loop = asyncio.get_event_loop() class MyProtocol(asyncio.Protocol): - transport = None + + def __init__(self, loop): + self.transport = None + self.on_con_lost = loop.create_future() def connection_made(self, transport): self.transport = transport @@ -795,35 +932,102 @@ the event loop :: def data_received(self, data): print("Received:", data.decode()) - # We are done: close the transport (it will call connection_lost()) + # We are done: close the transport; + # connection_lost() will be called automatically. self.transport.close() def connection_lost(self, exc): - # The socket has been closed, stop the event loop - loop.stop() + # The socket has been closed + self.on_con_lost.set_result(True) + + + async def main(): + # Get a reference to the event loop as we plan to use + # low-level APIs. + loop = asyncio.get_running_loop() - # Register the socket to wait for data - connect_coro = loop.create_connection(MyProtocol, sock=rsock) - transport, protocol = loop.run_until_complete(connect_coro) + # Create a pair of connected sockets + rsock, wsock = socket.socketpair() - # Simulate the reception of data from the network - loop.call_soon(wsock.send, 'abc'.encode()) + # Register the socket to wait for data. + transport, protocol = await loop.create_connection( + lambda: MyProtocol(loop), sock=rsock) - # Run the event loop - loop.run_forever() + # Simulate the reception of data from the network. + loop.call_soon(wsock.send, 'abc'.encode()) - # We are done, close sockets and the event loop - rsock.close() - wsock.close() - loop.close() + try: + await protocol.on_con_lost + finally: + transport.close() + wsock.close() + + asyncio.run(main()) .. seealso:: The :ref:`watch a file descriptor for read events ` example uses the low-level - :meth:`AbstractEventLoop.add_reader` method to register the file descriptor of a - socket. + :meth:`loop.add_reader` method to register an FD. The :ref:`register an open socket to wait for data using streams ` example uses high-level streams created by the :func:`open_connection` function in a coroutine. + +.. _asyncio-subprocess-proto-example: + +loop.subprocess_exec() and SubprocessProtocol +--------------------------------------------- + +An example of a subprocess protocol using to get the output of a +subprocess and to wait for the subprocess exit. + +The subprocess is created by th :meth:`loop.subprocess_exec` method:: + + import asyncio + import sys + + class DateProtocol(asyncio.SubprocessProtocol): + def __init__(self, exit_future): + self.exit_future = exit_future + self.output = bytearray() + + def pipe_data_received(self, fd, data): + self.output.extend(data) + + def process_exited(self): + self.exit_future.set_result(True) + + async def get_date(): + # Get a reference to the event loop as we plan to use + # low-level APIs. + loop = asyncio.get_running_loop() + + code = 'import datetime; print(datetime.datetime.now())' + exit_future = asyncio.Future(loop=loop) + + # Create the subprocess controlled by DateProtocol; + # redirect the standard output into a pipe. + transport, protocol = await loop.subprocess_exec( + lambda: DateProtocol(exit_future), + sys.executable, '-c', code, + stdin=None, stderr=None) + + # Wait for the subprocess exit using the process_exited() + # method of the protocol. + await exit_future + + # Close the stdout pipe. + transport.close() + + # Read the output which was collected by the + # pipe_data_received() method of the protocol. + data = bytes(protocol.output) + return data.decode('ascii').rstrip() + + if sys.platform == "win32": + asyncio.set_event_loop_policy( + asyncio.WindowsProactorEventLoopPolicy()) + + date = asyncio.run(get_date()) + print(f"Current date: {date}") diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 65497f29d895..1e4470adee98 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -1,41 +1,41 @@ .. currentmodule:: asyncio + +====== Queues ====== -**Source code:** :source:`Lib/asyncio/queues.py` - -Queues: +asyncio queues are designed to be similar to classes of the +:mod:`queue` module. Although asyncio queues are not thread-safe, +they are designed to be used specifically in async/await code. -* :class:`Queue` -* :class:`PriorityQueue` -* :class:`LifoQueue` +Note that methods on asyncio queues don't have a *timeout* parameter; +use :func:`asyncio.wait_for` function to do queue operations with a +timeout. -asyncio queue API was designed to be close to classes of the :mod:`queue` -module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, -:class:`~queue.LifoQueue`), but it has no *timeout* parameter. The -:func:`asyncio.wait_for` function can be used to cancel a task after a timeout. +See also the `Examples`_ section below. Queue ------ +===== .. class:: Queue(maxsize=0, \*, loop=None) - A queue, useful for coordinating producer and consumer coroutines. + A first in, first out (FIFO) queue. - If *maxsize* is less than or equal to zero, the queue size is infinite. If - it is an integer greater than ``0``, then ``await put()`` will block - when the queue reaches *maxsize*, until an item is removed by :meth:`get`. + If *maxsize* is less than or equal to zero, the queue size is + infinite. If it is an integer greater than ``0``, then + ``await put()`` blocks when the queue reaches *maxsize* + until an item is removed by :meth:`get`. - Unlike the standard library :mod:`queue`, you can reliably know this Queue's - size with :meth:`qsize`, since your single-threaded asyncio application won't - be interrupted between calling :meth:`qsize` and doing an operation on the - Queue. + Unlike the standard library threading :mod:`queue`, the size of + the queue is always known and can be returned by calling the + :meth:`qsize` method. This class is :ref:`not thread safe `. - .. versionchanged:: 3.4.4 - New :meth:`join` and :meth:`task_done` methods. + .. attribute:: maxsize + + Number of items allowed in the queue. .. method:: empty() @@ -45,26 +45,16 @@ Queue Return ``True`` if there are :attr:`maxsize` items in the queue. - .. note:: - - If the Queue was initialized with ``maxsize=0`` (the default), then - :meth:`full()` is never ``True``. + If the queue was initialized with ``maxsize=0`` (the default), + then :meth:`full()` never returns ``True``. .. coroutinemethod:: get() - Remove and return an item from the queue. If queue is empty, wait until - an item is available. - - This method is a :ref:`coroutine `. - - .. seealso:: - - The :meth:`empty` method. + Remove and return an item from the queue. If queue is empty, + wait until an item is available. .. method:: get_nowait() - Remove and return an item from the queue. - Return an item if one is immediately available, else raise :exc:`QueueEmpty`. @@ -72,26 +62,16 @@ Queue Block until all items in the queue have been gotten and processed. - The count of unfinished tasks goes up whenever an item is added to the - queue. The count goes down whenever a consumer thread calls - :meth:`task_done` to indicate that the item was retrieved and all work on - it is complete. When the count of unfinished tasks drops to zero, - :meth:`join` unblocks. - - This method is a :ref:`coroutine `. - - .. versionadded:: 3.4.4 + The count of unfinished tasks goes up whenever an item is added + to the queue. The count goes down whenever a consumer thread calls + :meth:`task_done` to indicate that the item was retrieved and all + work on it is complete. When the count of unfinished tasks drops + to zero, :meth:`join` unblocks. .. coroutinemethod:: put(item) - Put an item into the queue. If the queue is full, wait until a free slot - is available before adding item. - - This method is a :ref:`coroutine `. - - .. seealso:: - - The :meth:`full` method. + Put an item into the queue. If the queue is full, wait until a + free slot is available before adding item. .. method:: put_nowait(item) @@ -107,54 +87,111 @@ Queue Indicate that a formerly enqueued task is complete. - Used by queue consumers. For each :meth:`~Queue.get` used to fetch a task, a - subsequent call to :meth:`task_done` tells the queue that the processing - on the task is complete. - - If a :meth:`join` is currently blocking, it will resume when all items - have been processed (meaning that a :meth:`task_done` call was received - for every item that had been :meth:`~Queue.put` into the queue). + Used by queue consumers. For each :meth:`~Queue.get` used to + fetch a task, a subsequent call to :meth:`task_done` tells the + queue that the processing on the task is complete. - Raises :exc:`ValueError` if called more times than there were items - placed in the queue. + If a :meth:`join` is currently blocking, it will resume when all + items have been processed (meaning that a :meth:`task_done` + call was received for every item that had been :meth:`~Queue.put` + into the queue). - .. versionadded:: 3.4.4 + Raises :exc:`ValueError` if called more times than there were + items placed in the queue. - .. attribute:: maxsize - - Number of items allowed in the queue. - -PriorityQueue -------------- +Priority Queue +============== .. class:: PriorityQueue - A subclass of :class:`Queue`; retrieves entries in priority order (lowest - first). + A variant of :class:`Queue`; retrieves entries in priority order + (lowest first). - Entries are typically tuples of the form: (priority number, data). + Entries are typically tuples of the form + ``(priority_number, data)``. -LifoQueue ---------- +LIFO Queue +========== .. class:: LifoQueue - A subclass of :class:`Queue` that retrieves most recently added entries - first. + A variant of :class:`Queue` that retrieves most recently added + entries first (last in, first out). Exceptions -^^^^^^^^^^ +========== .. exception:: QueueEmpty - Exception raised when the :meth:`~Queue.get_nowait` method is called on a - :class:`Queue` object which is empty. + This exception is raised when the :meth:`~Queue.get_nowait` method + is called on an empty queue. .. exception:: QueueFull - Exception raised when the :meth:`~Queue.put_nowait` method is called on a - :class:`Queue` object which is full. + Exception raised when the :meth:`~Queue.put_nowait` method is called + on a queue that has reached its *maxsize*. + + +Examples +======== + +Queues can be used to distribute workload between several +concurrent tasks:: + + import asyncio + import random + import time + + + async def worker(name, queue): + while True: + # Get a "work item" out of the queue. + sleep_for = await queue.get() + + # Sleep for the "sleep_for" seconds. + await asyncio.sleep(sleep_for) + + # Notify the queue that the "work item" has been processed. + queue.task_done() + + print(f'{name} has slept for {sleep_for:.2f} seconds') + + + async def main(): + # Create a queue that we will use to store our "workload". + queue = asyncio.Queue() + + # Generate random timings and put them into the queue. + total_sleep_time = 0 + for _ in range(20): + sleep_for = random.uniform(0.05, 1.0) + total_sleep_time += sleep_for + queue.put_nowait(sleep_for) + + # Create three worker tasks to process the queue concurrently. + tasks = [] + for i in range(3): + task = asyncio.create_task(worker(f'worker-{i}', queue)) + tasks.append(task) + + # Wait until the queue is fully processed. + started_at = time.monotonic() + await queue.join() + total_slept_for = time.monotonic() - started_at + + # Cancel our worker tasks. + for task in tasks: + task.cancel() + # Wait until all worker tasks are cancelled. + await asyncio.gather(*tasks, return_exceptions=True) + + print('====') + print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds') + print(f'total expected sleep time: {total_sleep_time:.2f} seconds') + + + asyncio.run(main()) diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index f662e7223337..3fe7ac7b4abc 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -2,83 +2,107 @@ .. _asyncio-streams: -+++++++++++++++++++++++++++++ -Streams (coroutine based API) -+++++++++++++++++++++++++++++ +======= +Streams +======= -**Source code:** :source:`Lib/asyncio/streams.py` +Streams are high-level async/await-ready primitives to work with +network connections. Streams allow send and receive data without +using callbacks or low-level protocols and transports. -Stream functions -================ +Here's an example of a TCP echo client written using asyncio +streams:: -.. note:: + import asyncio + + async def tcp_echo_client(message): + reader, writer = await asyncio.open_connection( + '127.0.0.1', 8888) + + print(f'Send: {message!r}') + writer.write(message.encode()) + + data = await reader.read(100) + print(f'Received: {data.decode()!r}') + + print('Close the connection') + writer.close() - The top-level functions in this module are meant as convenience wrappers - only; there's really nothing special there, and if they don't do - exactly what you want, feel free to copy their code. + asyncio.run(tcp_echo_client('Hello World!')) -.. coroutinefunction:: open_connection(host=None, port=None, \*, loop=None, limit=None, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None, ssl_handshake_timeout=None) +.. rubric:: Stream Functions - A wrapper for :meth:`~AbstractEventLoop.create_connection()` returning a (reader, - writer) pair. +The following top-level asyncio functions can be used to create +and work with streams: - The reader returned is a :class:`StreamReader` instance; the writer is - a :class:`StreamWriter` instance. - When specified, the *loop* argument determines which event loop to use, - and the *limit* argument determines the buffer size limit used by the +.. coroutinefunction:: open_connection(host=None, port=None, \*, \ + loop=None, limit=None, ssl=None, family=0, \ + proto=0, flags=0, sock=None, local_addr=None, \ + server_hostname=None, ssl_handshake_timeout=None) + + Establish a network connection and return a pair of + ``(reader, writer)``. + + The returned *reader* and *writer* objects are instances of + :class:`StreamReader` and :class:`StreamWriter` classes. + + The *loop* argument is optional and can always be determined + automatically when this method is awaited from a coroutine. + + *limit* determines the buffer size limit used by the returned :class:`StreamReader` instance. The rest of the arguments are passed directly to - :meth:`AbstractEventLoop.create_connection`. - - This function is a :ref:`coroutine `. + :meth:`loop.create_connection`. .. versionadded:: 3.7 The *ssl_handshake_timeout* parameter. -.. coroutinefunction:: start_server(client_connected_cb, host=None, port=None, \*, loop=None, limit=None, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, start_serving=True) +.. coroutinefunction:: start_server(client_connected_cb, host=None, \ + port=None, \*, loop=None, limit=None, \ + family=socket.AF_UNSPEC, \ + flags=socket.AI_PASSIVE, sock=None, \ + backlog=100, ssl=None, reuse_address=None, \ + reuse_port=None, ssl_handshake_timeout=None, \ + start_serving=True) - Start a socket server, with a callback for each client connected. The return - value is the same as :meth:`~AbstractEventLoop.create_server()`. + Start a socket server. The *client_connected_cb* callback is called whenever a new client - connection is established. It receives a reader/writer pair as two - arguments, the first is a :class:`StreamReader` instance, - and the second is a :class:`StreamWriter` instance. + connection is established. It receives a ``(reader, writer)`` pair + as two arguments, instances of the :class:`StreamReader` and + :class:`StreamWriter` classes. - *client_connected_cb* accepts a plain callable or a + *client_connected_cb* can be a plain callable or a :ref:`coroutine function `; if it is a coroutine function, - it will be automatically converted into a :class:`Task`. + it will be automatically wrapped into a :class:`Task`. - When specified, the *loop* argument determines which event loop to use, - and the *limit* argument determines the buffer size limit used by the - :class:`StreamReader` instance passed to *client_connected_cb*. + The *loop* argument is optional and can always be determined + automatically when this method is awaited from a coroutine. - The rest of the arguments are passed directly to - :meth:`~AbstractEventLoop.create_server()`. + *limit* determines the buffer size limit used by the + returned :class:`StreamReader` instance. - This function is a :ref:`coroutine `. + The rest of the arguments are passed directly to + :meth:`loop.create_server`. .. versionadded:: 3.7 The *ssl_handshake_timeout* and *start_serving* parameters. -.. coroutinefunction:: open_unix_connection(path=None, \*, loop=None, limit=None, ssl=None, sock=None, server_hostname=None, ssl_handshake_timeout=None) - - A wrapper for :meth:`~AbstractEventLoop.create_unix_connection()` returning - a (reader, writer) pair. +.. coroutinefunction:: open_unix_connection(path=None, \*, loop=None, \ + limit=None, ssl=None, sock=None, \ + server_hostname=None, ssl_handshake_timeout=None) - When specified, the *loop* argument determines which event loop to use, - and the *limit* argument determines the buffer size limit used by the - returned :class:`StreamReader` instance. + Establish a UNIX socket connection and return a pair of + ``(reader, writer)``. - The rest of the arguments are passed directly to - :meth:`~AbstractEventLoop.create_unix_connection()`. + Similar to :func:`open_connection` but operates on UNIX sockets. - This function is a :ref:`coroutine `. + See also the documentation of :meth:`loop.create_unix_connection`. Availability: UNIX. @@ -90,27 +114,16 @@ Stream functions The *path* parameter can now be a :term:`path-like object` -.. coroutinefunction:: start_unix_server(client_connected_cb, path=None, \*, loop=None, limit=None, sock=None, backlog=100, ssl=None, ssl_handshake_timeout=None, start_serving=True) - - Start a UNIX Domain Socket server, with a callback for each client connected. - - The *client_connected_cb* callback is called whenever a new client - connection is established. It receives a reader/writer pair as two - arguments, the first is a :class:`StreamReader` instance, - and the second is a :class:`StreamWriter` instance. - - *client_connected_cb* accepts a plain callable or a - :ref:`coroutine function `; if it is a coroutine function, - it will be automatically converted into a :class:`Task`. +.. coroutinefunction:: start_unix_server(client_connected_cb, path=None, \ + \*, loop=None, limit=None, sock=None, \ + backlog=100, ssl=None, ssl_handshake_timeout=None, \ + start_serving=True) - When specified, the *loop* argument determines which event loop to use, - and the *limit* argument determines the buffer size limit used by the - :class:`StreamReader` instance passed to *client_connected_cb*. + Start a UNIX socket server. - The rest of the arguments are passed directly to - :meth:`~AbstractEventLoop.create_unix_server()`. + Similar to :func:`start_server` but operates on UNIX sockets. - This function is a :ref:`coroutine `. + See also the documentation of :meth:`loop.create_unix_server`. Availability: UNIX. @@ -123,6 +136,13 @@ Stream functions The *path* parameter can now be a :term:`path-like object`. +.. rubric:: Contents + +* `StreamReader`_ and `StreamWriter`_ +* `StreamReaderProtocol`_ +* `Examples`_ + + StreamReader ============ @@ -159,8 +179,6 @@ StreamReader If the EOF was received and the internal buffer is empty, return an empty ``bytes`` object. - This method is a :ref:`coroutine `. - .. coroutinemethod:: readline() Read one line, where "line" is a sequence of bytes ending with ``\n``. @@ -171,8 +189,6 @@ StreamReader If the EOF was received and the internal buffer is empty, return an empty ``bytes`` object. - This method is a :ref:`coroutine `. - .. coroutinemethod:: readexactly(n) Read exactly *n* bytes. Raise an :exc:`IncompleteReadError` if the end of @@ -180,8 +196,6 @@ StreamReader :attr:`IncompleteReadError.partial` attribute of the exception contains the partial read bytes. - This method is a :ref:`coroutine `. - .. coroutinemethod:: readuntil(separator=b'\\n') Read data from the stream until ``separator`` is found. @@ -208,7 +222,8 @@ StreamReader .. method:: at_eof() - Return ``True`` if the buffer is empty and :meth:`feed_eof` was called. + Return ``True`` if the buffer is empty and :meth:`feed_eof` + was called. StreamWriter @@ -299,7 +314,8 @@ StreamWriter StreamReaderProtocol ==================== -.. class:: StreamReaderProtocol(stream_reader, client_connected_cb=None, loop=None) +.. class:: StreamReaderProtocol(stream_reader, client_connected_cb=None, \ + loop=None) Trivial helper class to adapt between :class:`Protocol` and :class:`StreamReader`. Subclass of :class:`Protocol`. @@ -314,36 +330,8 @@ StreamReaderProtocol accidentally calling inappropriate methods of the protocol.) -IncompleteReadError -=================== - -.. exception:: IncompleteReadError - - Incomplete read error, subclass of :exc:`EOFError`. - - .. attribute:: expected - - Total number of expected bytes (:class:`int`). - - .. attribute:: partial - - Read bytes string before the end of stream was reached (:class:`bytes`). - - -LimitOverrunError -================= - -.. exception:: LimitOverrunError - - Reached the buffer limit while looking for a separator. - - .. attribute:: consumed - - Total number of to be consumed bytes. - - -Stream examples -=============== +Examples +======== .. _asyncio-tcp-echo-client-streams: @@ -354,28 +342,26 @@ TCP echo client using the :func:`asyncio.open_connection` function:: import asyncio - async def tcp_echo_client(message, loop): - reader, writer = await asyncio.open_connection('127.0.0.1', 8888, - loop=loop) + async def tcp_echo_client(message): + reader, writer = await asyncio.open_connection( + '127.0.0.1', 8888) - print('Send: %r' % message) + print(f'Send: {message!r}') writer.write(message.encode()) data = await reader.read(100) - print('Received: %r' % data.decode()) + print(f'Received: {data.decode()!r}') - print('Close the socket') + print('Close the connection') writer.close() - message = 'Hello World!' - loop = asyncio.get_event_loop() - loop.run_until_complete(tcp_echo_client(message, loop)) - loop.close() + asyncio.run(tcp_echo_client('Hello World!')) + .. seealso:: The :ref:`TCP echo client protocol ` - example uses the :meth:`AbstractEventLoop.create_connection` method. + example uses the low-level :meth:`loop.create_connection` method. .. _asyncio-tcp-echo-server-streams: @@ -391,35 +377,33 @@ TCP echo server using the :func:`asyncio.start_server` function:: data = await reader.read(100) message = data.decode() addr = writer.get_extra_info('peername') - print("Received %r from %r" % (message, addr)) - print("Send: %r" % message) + print(f"Received {message!r} from {addr!r}") + + print(f"Send: {message!r}") writer.write(data) await writer.drain() - print("Close the client socket") + print("Close the connection") writer.close() - loop = asyncio.get_event_loop() - coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop) - server = loop.run_until_complete(coro) + async def main(): + server = await asyncio.start_server( + handle_echo, '127.0.0.1', 8888) + + addr = server.sockets[0].getsockname() + print(f'Serving on {addr}') - # Serve requests until Ctrl+C is pressed - print('Serving on {}'.format(server.sockets[0].getsockname())) - try: - loop.run_forever() - except KeyboardInterrupt: - pass + async with server: + await server.serve_forever() + + asyncio.run(main()) - # Close the server - server.close() - loop.run_until_complete(server.wait_closed()) - loop.close() .. seealso:: The :ref:`TCP echo server protocol ` - example uses the :meth:`AbstractEventLoop.create_server` method. + example uses the :meth:`loop.create_server` method. Get HTTP headers @@ -434,30 +418,34 @@ Simple example querying HTTP headers of the URL passed on the command line:: async def print_http_headers(url): url = urllib.parse.urlsplit(url) if url.scheme == 'https': - connect = asyncio.open_connection(url.hostname, 443, ssl=True) + reader, writer = await asyncio.open_connection( + url.hostname, 443, ssl=True) else: - connect = asyncio.open_connection(url.hostname, 80) - reader, writer = await connect - query = ('HEAD {path} HTTP/1.0\r\n' - 'Host: {hostname}\r\n' - '\r\n').format(path=url.path or '/', hostname=url.hostname) + reader, writer = await asyncio.open_connection( + url.hostname, 80) + + query = ( + f"HEAD {url.path or '/'} HTTP/1.0\r\n" + f"Host: {url.hostname}\r\n" + f"\r\n" + ) + writer.write(query.encode('latin-1')) while True: line = await reader.readline() if not line: break + line = line.decode('latin1').rstrip() if line: - print('HTTP header> %s' % line) + print(f'HTTP header> {line}') # Ignore the body, close the socket writer.close() url = sys.argv[1] - loop = asyncio.get_event_loop() - task = asyncio.ensure_future(print_http_headers(url)) - loop.run_until_complete(task) - loop.close() + asyncio.run(print_http_headers(url)) + Usage:: @@ -467,6 +455,7 @@ or with HTTPS:: python example.py https://example.com/path/page.html + .. _asyncio-register-socket-streams: Register an open socket to wait for data using streams @@ -476,14 +465,18 @@ Coroutine waiting until a socket receives data using the :func:`open_connection` function:: import asyncio - from socket import socketpair + import socket + + async def wait_for_data(): + # Get a reference to the current event loop because + # we want to access low-level APIs. + loop = asyncio.get_running_loop() - async def wait_for_data(loop): - # Create a pair of connected sockets - rsock, wsock = socketpair() + # Create a pair of connected sockets. + rsock, wsock = socket.socketpair() - # Register the open socket to wait for data - reader, writer = await asyncio.open_connection(sock=rsock, loop=loop) + # Register the open socket to wait for data. + reader, writer = await asyncio.open_connection(sock=rsock) # Simulate the reception of data from the network loop.call_soon(wsock.send, 'abc'.encode()) @@ -498,17 +491,14 @@ Coroutine waiting until a socket receives data using the # Close the second socket wsock.close() - loop = asyncio.get_event_loop() - loop.run_until_complete(wait_for_data(loop)) - loop.close() + asyncio.run(wait_for_data()) .. seealso:: The :ref:`register an open socket to wait for data using a protocol - ` example uses a low-level protocol created by the - :meth:`AbstractEventLoop.create_connection` method. + ` example uses a low-level protocol and + the :meth:`loop.create_connection` method. The :ref:`watch a file descriptor for read events ` example uses the low-level - :meth:`AbstractEventLoop.add_reader` method to register the file descriptor of a - socket. + :meth:`loop.add_reader` method to watch a file descriptor. diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index 60e174574b04..b05c236bb0c8 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -2,137 +2,91 @@ .. _asyncio-subprocess: -Subprocess -========== +============ +Subprocesses +============ -**Source code:** :source:`Lib/asyncio/subprocess.py` +This section describes high-level async/await asyncio APIs to +create and manage subprocesses. -Windows event loop ------------------- +Here's an example of how asyncio can run a shell command and +communicate its result back:: -On Windows, the default event loop is :class:`SelectorEventLoop` which does not -support subprocesses. :class:`ProactorEventLoop` should be used instead. -Example to use it on Windows:: - - import asyncio, sys - - if sys.platform == 'win32': - loop = asyncio.ProactorEventLoop() - asyncio.set_event_loop(loop) - -.. seealso:: - - :ref:`Available event loops ` and :ref:`Platform - support `. - - -Create a subprocess: high-level API using Process -------------------------------------------------- - -.. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) - - Create a subprocess. - - The *limit* parameter sets the buffer limit passed to the - :class:`StreamReader`. See :meth:`AbstractEventLoop.subprocess_exec` for other - parameters. - - Return a :class:`~asyncio.subprocess.Process` instance. - - This function is a :ref:`coroutine `. - -.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) - - Run the shell command *cmd*. - - The *limit* parameter sets the buffer limit passed to the - :class:`StreamReader`. See :meth:`AbstractEventLoop.subprocess_shell` for other - parameters. + import asyncio - Return a :class:`~asyncio.subprocess.Process` instance. + async def run(cmd): + proc = await asyncio.create_subprocess_shell( + cmd, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE) - It is the application's responsibility to ensure that all whitespace and - metacharacters are quoted appropriately to avoid `shell injection - `_ - vulnerabilities. The :func:`shlex.quote` function can be used to properly - escape whitespace and shell metacharacters in strings that are going to be - used to construct shell commands. + stdout, stderr = await proc.communicate() - This function is a :ref:`coroutine `. + print(f'[{cmd!r} exited with {proc.returncode}]') + if stdout: + print(f'[stdout]\n{stdout.decode()}') + if stderr: + print(f'[stderr]\n{stderr.decode()}') -Use the :meth:`AbstractEventLoop.connect_read_pipe` and -:meth:`AbstractEventLoop.connect_write_pipe` methods to connect pipes. + asyncio.run(run('ls /zzz')) +will print:: -Create a subprocess: low-level API using subprocess.Popen ---------------------------------------------------------- + ['ls /zzz' exited with 1] + [stderr] + ls: /zzz: No such file or directory -Run subprocesses asynchronously using the :mod:`subprocess` module. +Because all asyncio subprocess functions are asynchronous and asyncio +provides many tools to work with such functions, it is easy to execute +and monitor multiple subprocesses in parallel. It is indeed trivial +to modify the above example to run a few commands at once:: -.. coroutinemethod:: AbstractEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) + async def main(): + await asyncio.gather( + run('ls /zzz'), + run('sleep 1; echo "hello"')) - Create a subprocess from one or more string arguments (character strings or - bytes strings encoded to the :ref:`filesystem encoding - `), where the first string - specifies the program to execute, and the remaining strings specify the - program's arguments. (Thus, together the string arguments form the - ``sys.argv`` value of the program, assuming it is a Python script.) This is - similar to the standard library :class:`subprocess.Popen` class called with - shell=False and the list of strings passed as the first argument; - however, where :class:`~subprocess.Popen` takes a single argument which is - list of strings, :func:`subprocess_exec` takes multiple string arguments. + asyncio.run(main()) - The *protocol_factory* must instantiate a subclass of the - :class:`asyncio.SubprocessProtocol` class. +See also the `Examples`_ subsection. - Other parameters: - * *stdin*: Either a file-like object representing the pipe to be connected - to the subprocess's standard input stream using - :meth:`~AbstractEventLoop.connect_write_pipe`, or the constant - :const:`subprocess.PIPE` (the default). By default a new pipe will be - created and connected. +Creating Subprocesses +===================== - * *stdout*: Either a file-like object representing the pipe to be connected - to the subprocess's standard output stream using - :meth:`~AbstractEventLoop.connect_read_pipe`, or the constant - :const:`subprocess.PIPE` (the default). By default a new pipe will be - created and connected. +.. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, \ + stdout=None, stderr=None, loop=None, \ + limit=None, \*\*kwds) - * *stderr*: Either a file-like object representing the pipe to be connected - to the subprocess's standard error stream using - :meth:`~AbstractEventLoop.connect_read_pipe`, or one of the constants - :const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`. - By default a new pipe will be created and connected. When - :const:`subprocess.STDOUT` is specified, the subprocess's standard error - stream will be connected to the same pipe as the standard output stream. + Create a subprocess. - * All other keyword arguments are passed to :class:`subprocess.Popen` - without interpretation, except for *bufsize*, *universal_newlines* and - *shell*, which should not be specified at all. + The *limit* argument sets the buffer limit for :class:`StreamReader` + wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` + (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* + arguments). - Returns a pair of ``(transport, protocol)``, where *transport* is an - instance of :class:`BaseSubprocessTransport`. + Return a :class:`~asyncio.subprocess.Process` instance. - This method is a :ref:`coroutine `. + See the documentation of :meth:`loop.subprocess_exec` for other + parameters. - See the constructor of the :class:`subprocess.Popen` class for parameters. +.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \ + stdout=None, stderr=None, loop=None, \ + limit=None, \*\*kwds) -.. coroutinemethod:: AbstractEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs) + Run the shell command *cmd*. - Create a subprocess from *cmd*, which is a character string or a bytes - string encoded to the :ref:`filesystem encoding `, - using the platform's "shell" syntax. This is similar to the standard library - :class:`subprocess.Popen` class called with ``shell=True``. + The *limit* argument sets the buffer limit for :class:`StreamReader` + wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` + (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* + arguments). - The *protocol_factory* must instantiate a subclass of the - :class:`asyncio.SubprocessProtocol` class. + Return a :class:`~asyncio.subprocess.Process` instance. - See :meth:`~AbstractEventLoop.subprocess_exec` for more details about - the remaining arguments. + See the documentation of :meth:`loop.subprocess_shell` for other + parameters. - Returns a pair of ``(transport, protocol)``, where *transport* is an - instance of :class:`BaseSubprocessTransport`. +.. note:: It is the application's responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid `shell injection @@ -141,106 +95,125 @@ Run subprocesses asynchronously using the :mod:`subprocess` module. escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands. - This method is a :ref:`coroutine `. +.. note:: + + The default event loop that asyncio is pre-configured + to use on **Windows** does not support subprocesses. + See :ref:`Subprocess Support on Windows ` + for details. .. seealso:: - The :meth:`AbstractEventLoop.connect_read_pipe` and - :meth:`AbstractEventLoop.connect_write_pipe` methods. + asyncio has also *low-level* APIs to work with subprocesses: + :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`, + :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`, + as well as the :ref:`Subprocess Transports ` + and :ref:`Subprocess Protocols `. Constants ---------- +========= .. data:: asyncio.subprocess.PIPE - Special value that can be used as the *stdin*, *stdout* or *stderr* argument - to :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and - indicates that a pipe to the standard stream should be opened. + Can be passed to the *stdin*, *stdout* or *stderr* parameters. + + If *PIPE* is passed to *stdin* argument, the + :attr:`Process.stdin ` attribute + will point to a :class:`StreamWriter` instance. + + If *PIPE* is passed to *stdout* or *stderr* arguments, the + :attr:`Process.stdout ` and + :attr:`Process.stderr ` + attributes will point to :class:`StreamReader` instances. .. data:: asyncio.subprocess.STDOUT - Special value that can be used as the *stderr* argument to - :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and - indicates that standard error should go into the same handle as standard - output. + Can be passed to the *stderr* parameter to redirect process' + *stderr* to *stdout*. .. data:: asyncio.subprocess.DEVNULL - Special value that can be used as the *stdin*, *stdout* or *stderr* argument - to :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and - indicates that the special file :data:`os.devnull` will be used. + Can be passed as the *stdin*, *stdout* or *stderr* parameters + to redirect the corresponding subprocess' IO to :data:`os.devnull`. -Process -------- +Interacting with Subprocesses +============================= + +Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell` +functions return instances of the *Process* class. It is a high-level +wrapper that allows to watch for subprocesses completion and +communicate with them. .. class:: asyncio.subprocess.Process - A subprocess created by the :func:`create_subprocess_exec` or the - :func:`create_subprocess_shell` function. + An object that wraps OS processes created by the + :func:`create_subprocess_exec` and :func:`create_subprocess_shell` + functions. + + This class is designed to have a similar API to the + :class:`subprocess.Popen` class, but there are some + notable differences: + + * unlike Popen, Process instances do not have an equivalent to + the :meth:`~subprocess.Popen.poll` method; + + * the :meth:`~asyncio.subprocess.Process.communicate` and + :meth:`~asyncio.subprocess.Process.wait` methods don't take a + *timeout* parameter: use the :func:`wait_for` function; + + * the :meth:`Process.wait() ` method + is asynchronous, whereas :meth:`subprocess.Popen.wait` method + is implemented as a blocking busy loop; - The API of the :class:`~asyncio.subprocess.Process` class was designed to be - close to the API of the :class:`subprocess.Popen` class, but there are some - differences: + * the *universal_newlines* parameter is not supported. - * There is no explicit :meth:`~subprocess.Popen.poll` method - * The :meth:`~subprocess.Popen.communicate` and - :meth:`~subprocess.Popen.wait` methods don't take a *timeout* parameter: - use the :func:`wait_for` function - * The *universal_newlines* parameter is not supported (only bytes strings - are supported) - * The :meth:`~asyncio.subprocess.Process.wait` method of - the :class:`~asyncio.subprocess.Process` class is asynchronous whereas the - :meth:`~subprocess.Popen.wait` method of the :class:`~subprocess.Popen` - class is implemented as a busy loop. + This class is :ref:`not thread safe `. - This class is :ref:`not thread safe `. See also the - :ref:`Subprocess and threads ` section. + See also the :ref:`Subprocess and Threads ` + section. .. coroutinemethod:: wait() - Wait for child process to terminate. Set and return :attr:`returncode` - attribute. + Wait for child process to terminate. - This method is a :ref:`coroutine `. + Set and return the :attr:`returncode` attribute. .. note:: - This will deadlock when using ``stdout=PIPE`` or ``stderr=PIPE`` and - the child process generates enough output to a pipe such that it - blocks waiting for the OS pipe buffer to accept more data. Use the - :meth:`communicate` method when using pipes to avoid that. + This method can deadlock when using ``stdout=PIPE`` or + ``stderr=PIPE`` and the child process generates so much output + that it blocks waiting for the OS pipe buffer to accept + more data. Use the :meth:`communicate` method when using pipes + to avoid this condition. .. coroutinemethod:: communicate(input=None) - Interact with process: Send data to stdin. Read data from stdout and - stderr, until end-of-file is reached. Wait for process to terminate. - The optional *input* argument should be data to be sent to the child - process, or ``None``, if no data should be sent to the child. The type - of *input* must be bytes. + Interact with process: - :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. + 1. send data to *stdin* (if *input* is not ``None``); + 2. read data from *stdout* and *stderr*, until EOF is reached; + 3. wait for process to terminate. - If a :exc:`BrokenPipeError` or :exc:`ConnectionResetError` exception is - raised when writing *input* into stdin, the exception is ignored. It - occurs when the process exits before all data are written into stdin. + The optional *input* argument is the data (:class:`bytes` object) + that will be sent to the child process. - Note that if you want to send data to the process's stdin, you need to - create the Process object with ``stdin=PIPE``. Similarly, to get anything - other than ``None`` in the result tuple, you need to give ``stdout=PIPE`` - and/or ``stderr=PIPE`` too. + Return a tuple ``(stdout_data, stderr_data)``. - This method is a :ref:`coroutine `. + If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError` + exception is raised when writing *input* into *stdin*, the + exception is ignored. This condition occurs when the process + exits before all data are written into *stdin*. - .. note:: - - The data read is buffered in memory, so do not use this method if the - data size is large or unlimited. + If its desired to send data to the process' *stdin*, + the process needs to be created with ``stdin=PIPE``. Similarly, + to get anything other than ``None`` in the result tuple, the + process has to be created with ``stdout=PIPE`` and/or + ``stderr=PIPE`` arguments. - .. versionchanged:: 3.4.2 - The method now ignores :exc:`BrokenPipeError` and - :exc:`ConnectionResetError`. + Note, that the data read is buffered in memory, so do not use + this method if the data size is large or unlimited. .. method:: send_signal(signal) @@ -255,67 +228,81 @@ Process .. method:: terminate() - Stop the child. On Posix OSs the method sends :py:data:`signal.SIGTERM` - to the child. On Windows the Win32 API function - :c:func:`TerminateProcess` is called to stop the child. + Stop the child. + + On Posix OSs the method sends :py:data:`signal.SIGTERM` to the + child process. + + On Windows the Win32 API function :c:func:`TerminateProcess` is + called to stop the child process. .. method:: kill() - Kills the child. On Posix OSs the function sends :py:data:`SIGKILL` to - the child. On Windows :meth:`kill` is an alias for :meth:`terminate`. + Kill the child. + + On Posix OSs the function sends :py:data:`SIGKILL` to the child + process. + + On Windows this method is an alias for :meth:`terminate`. .. attribute:: stdin - Standard input stream (:class:`StreamWriter`), ``None`` if the process - was created with ``stdin=None``. + Standard input stream (:class:`StreamWriter`) or ``None`` + if the process was created with ``stdin=None``. .. attribute:: stdout - Standard output stream (:class:`StreamReader`), ``None`` if the process - was created with ``stdout=None``. + Standard output stream (:class:`StreamReader`) or ``None`` + if the process was created with ``stdout=None``. .. attribute:: stderr - Standard error stream (:class:`StreamReader`), ``None`` if the process - was created with ``stderr=None``. + Standard error stream (:class:`StreamReader`) or ``None`` + if the process was created with ``stderr=None``. .. warning:: - Use the :meth:`communicate` method rather than :attr:`.stdin.write - `, :attr:`.stdout.read ` or :attr:`.stderr.read ` - to avoid deadlocks due to streams pausing reading or writing and blocking - the child process. + Use the :meth:`communicate` method rather than + :attr:`process.stdin.write() `, + :attr:`await process.stdout.read() ` or + :attr:`await process.stderr.read ` + to avoid deadlocks due to streams pausing reading or writing + and blocking the child process. .. attribute:: pid - The identifier of the process. + Process identification number (PID). Note that for processes created by the :func:`create_subprocess_shell` - function, this attribute is the process identifier of the spawned shell. + function, this attribute is the PID of the spawned shell. .. attribute:: returncode - Return code of the process when it exited. A ``None`` value indicates - that the process has not terminated yet. + Return code of the process when it exits. + + A ``None`` value indicates that the process has not terminated yet. - A negative value ``-N`` indicates that the child was terminated by signal - ``N`` (Unix only). + A negative value ``-N`` indicates that the child was terminated + by signal ``N`` (Unix only). .. _asyncio-subprocess-threads: -Subprocess and threads +Subprocess and Threads ---------------------- -asyncio supports running subprocesses from different threads, but there -are limits: +asyncio built-in event loops support running subprocesses from +different threads, but there are the following limitations: -* An event loop must run in the main thread -* The child watcher must be instantiated in the main thread, before executing - subprocesses from other threads. Call the :func:`get_child_watcher` - function in the main thread to instantiate the child watcher. +* An event loop must run in the main thread. -The :class:`asyncio.subprocess.Process` class is not thread safe. +* The child watcher must be instantiated in the main thread, + before executing subprocesses from other threads. Call the + :func:`get_child_watcher` function in the main thread to instantiate + the child watcher. + +Note, that alternative event loop implementations might not share +the above limitations; please refer to their documentation. .. seealso:: @@ -323,97 +310,43 @@ The :class:`asyncio.subprocess.Process` class is not thread safe. ` section. -Subprocess examples -------------------- - -Subprocess using transport and protocol -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Example of a subprocess protocol using to get the output of a subprocess and to -wait for the subprocess exit. The subprocess is created by the -:meth:`AbstractEventLoop.subprocess_exec` method:: - - import asyncio - import sys - - class DateProtocol(asyncio.SubprocessProtocol): - def __init__(self, exit_future): - self.exit_future = exit_future - self.output = bytearray() - - def pipe_data_received(self, fd, data): - self.output.extend(data) - - def process_exited(self): - self.exit_future.set_result(True) - - async def get_date(loop): - code = 'import datetime; print(datetime.datetime.now())' - exit_future = asyncio.Future(loop=loop) - - # Create the subprocess controlled by the protocol DateProtocol, - # redirect the standard output into a pipe - transport, protocol = await loop.subprocess_exec( - lambda: DateProtocol(exit_future), - sys.executable, '-c', code, - stdin=None, stderr=None) - - # Wait for the subprocess exit using the process_exited() method - # of the protocol - await exit_future - - # Close the stdout pipe - transport.close() +Examples +-------- - # Read the output which was collected by the pipe_data_received() - # method of the protocol - data = bytes(protocol.output) - return data.decode('ascii').rstrip() +An example using the :class:`~asyncio.subprocess.Process` class to +control a subprocess and the :class:`StreamReader` class to read from +the *stdout*. - if sys.platform == "win32": - loop = asyncio.ProactorEventLoop() - asyncio.set_event_loop(loop) - else: - loop = asyncio.get_event_loop() - - date = loop.run_until_complete(get_date(loop)) - print("Current date: %s" % date) - loop.close() - - -Subprocess using streams -^^^^^^^^^^^^^^^^^^^^^^^^ - -Example using the :class:`~asyncio.subprocess.Process` class to control the -subprocess and the :class:`StreamReader` class to read from the standard -output. The subprocess is created by the :func:`create_subprocess_exec` +The subprocess is created by the :func:`create_subprocess_exec` function:: - import asyncio.subprocess + import asyncio import sys async def get_date(): code = 'import datetime; print(datetime.datetime.now())' - # Create the subprocess, redirect the standard output into a pipe + # Create the subprocess; redirect the standard output + # into a pipe. proc = await asyncio.create_subprocess_exec( sys.executable, '-c', code, stdout=asyncio.subprocess.PIPE) - # Read one line of output + # Read one line of output. data = await proc.stdout.readline() line = data.decode('ascii').rstrip() - # Wait for the subprocess exit + # Wait for the subprocess exit. await proc.wait() return line if sys.platform == "win32": - loop = asyncio.ProactorEventLoop() - asyncio.set_event_loop(loop) - else: - loop = asyncio.get_event_loop() - - date = loop.run_until_complete(get_date()) - print("Current date: %s" % date) - loop.close() + asyncio.set_event_loop_policy( + asyncio.WindowsProactorEventLoopPolicy()) + + date = asyncio.run(get_date()) + print(f"Current date: {date}") + + +See also the :ref:`same example ` +written using low-level APIs. diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index c73f36b1adb9..9f4433ddc339 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -63,7 +63,7 @@ the coroutine object returned by the call doesn't do anything until you schedule its execution. There are two basic ways to start it running: call ``await coroutine`` or ``yield from coroutine`` from another coroutine (assuming the other coroutine is already running!), or schedule its execution -using the :func:`ensure_future` function or the :meth:`AbstractEventLoop.create_task` +using the :func:`ensure_future` function or the :meth:`loop.create_task` method. @@ -129,7 +129,7 @@ Example of coroutine displaying ``"Hello World"``:: .. seealso:: The :ref:`Hello World with call_soon() ` - example uses the :meth:`AbstractEventLoop.call_soon` method to schedule a + example uses the :meth:`loop.call_soon` method to schedule a callback. @@ -159,7 +159,7 @@ using the :meth:`sleep` function:: The :ref:`display the current date with call_later() ` example uses a callback with the - :meth:`AbstractEventLoop.call_later` method. + :meth:`loop.call_later` method. Example: Chain coroutines @@ -190,32 +190,12 @@ Sequence diagram of the example: .. image:: tulip_coro.png :align: center -The "Task" is created by the :meth:`AbstractEventLoop.run_until_complete` method +The "Task" is created by the :meth:`loop.run_until_complete` method when it gets a coroutine object instead of a task. The diagram shows the control flow, it does not describe exactly how things work internally. For example, the sleep coroutine creates an internal future -which uses :meth:`AbstractEventLoop.call_later` to wake up the task in 1 second. - - -InvalidStateError ------------------ - -.. exception:: InvalidStateError - - The operation is not allowed in this state. - - -TimeoutError ------------- - -.. exception:: TimeoutError - - The operation exceeded the given deadline. - -.. note:: - - This exception is different from the builtin :exc:`TimeoutError` exception! +which uses :meth:`loop.call_later` to wake up the task in 1 second. Future @@ -231,7 +211,7 @@ Future raise an exception when the future isn't done yet. - Callbacks registered with :meth:`add_done_callback` are always called - via the event loop's :meth:`~AbstractEventLoop.call_soon`. + via the event loop's :meth:`loop.call_soon`. - This class is not compatible with the :func:`~concurrent.futures.wait` and :func:`~concurrent.futures.as_completed` functions in the @@ -281,7 +261,7 @@ Future The *callback* is called with a single argument - the future object. If the future is already done when this is called, the callback is scheduled - with :meth:`~AbstractEventLoop.call_soon`. + with :meth:`loop.call_soon`. An optional keyword-only *context* argument allows specifying a custom :class:`contextvars.Context` for the *callback* to run in. The current @@ -344,11 +324,11 @@ Example combining a :class:`Future` and a :ref:`coroutine function The coroutine function is responsible for the computation (which takes 1 second) and it stores the result into the future. The -:meth:`~AbstractEventLoop.run_until_complete` method waits for the completion of +:meth:`loop.run_until_complete` method waits for the completion of the future. .. note:: - The :meth:`~AbstractEventLoop.run_until_complete` method uses internally the + The :meth:`loop.run_until_complete` method uses internally the :meth:`~Future.add_done_callback` method to be notified when the future is done. @@ -433,7 +413,7 @@ Task logged: see :ref:`Pending task destroyed `. Don't directly create :class:`Task` instances: use the :func:`create_task` - function or the :meth:`AbstractEventLoop.create_task` method. + function or the :meth:`loop.create_task` method. Tasks support the :mod:`contextvars` module. When a Task is created it copies the current context and later runs its coroutine @@ -644,7 +624,7 @@ Task functions .. seealso:: The :func:`create_task` function and - :meth:`AbstractEventLoop.create_task` method. + :meth:`loop.create_task` method. .. function:: wrap_future(future, \*, loop=None) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst index b076b7d0093b..0d58a94c8b9b 100644 --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -50,7 +50,8 @@ Table of contents: :maxdepth: 3 asyncio-eventloop.rst - asyncio-eventloops.rst + asyncio-policy.rst + asyncio-platforms.rst asyncio-task.rst asyncio-protocol.rst asyncio-stream.rst @@ -58,6 +59,7 @@ Table of contents: asyncio-sync.rst asyncio-queue.rst asyncio-dev.rst + asyncio-exceptions.rst .. seealso:: diff --git a/Doc/library/ipc.rst b/Doc/library/ipc.rst index 6b1756331ec0..8f5b3b216c78 100644 --- a/Doc/library/ipc.rst +++ b/Doc/library/ipc.rst @@ -16,11 +16,11 @@ The list of modules described in this chapter is: .. toctree:: + asyncio.rst socket.rst ssl.rst select.rst selectors.rst - asyncio.rst asyncore.rst asynchat.rst signal.rst diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst index 96f86e6c6142..d3aed84d250e 100644 --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -777,18 +777,18 @@ all changes introduced in Python 3.5 have also been backported to Python 3.4.x. Notable changes in the :mod:`asyncio` module since Python 3.4.0: -* New debugging APIs: :meth:`loop.set_debug() ` - and :meth:`loop.get_debug() ` methods. +* New debugging APIs: :meth:`loop.set_debug() ` + and :meth:`loop.get_debug() ` methods. (Contributed by Victor Stinner.) * The proactor event loop now supports SSL. (Contributed by Antoine Pitrou and Victor Stinner in :issue:`22560`.) -* A new :meth:`loop.is_closed() ` method to +* A new :meth:`loop.is_closed() ` method to check if the event loop is closed. (Contributed by Victor Stinner in :issue:`21326`.) -* A new :meth:`loop.create_task() ` +* A new :meth:`loop.create_task() ` to conveniently create and schedule a new :class:`~asyncio.Task` for a coroutine. The ``create_task`` method is also used by all asyncio functions that wrap coroutines into tasks, such as @@ -805,10 +805,10 @@ Notable changes in the :mod:`asyncio` module since Python 3.4.0: (Contributed by Yury Selivanov.) * New :meth:`loop.set_task_factory() - ` and - :meth:`loop.get_task_factory() ` + ` and + :meth:`loop.get_task_factory() ` methods to customize the task factory that :meth:`loop.create_task() - ` method uses. (Contributed by Yury + ` method uses. (Contributed by Yury Selivanov.) * New :meth:`Queue.join() ` and @@ -822,7 +822,7 @@ Notable changes in the :mod:`asyncio` module since Python 3.4.0: Updates in 3.5.1: * The :func:`~asyncio.ensure_future` function and all functions that - use it, such as :meth:`loop.run_until_complete() `, + use it, such as :meth:`loop.run_until_complete() `, now accept all kinds of :term:`awaitable objects `. (Contributed by Yury Selivanov.) @@ -834,20 +834,20 @@ Updates in 3.5.1: method to check if the transport is closing or closed. (Contributed by Yury Selivanov.) -* The :meth:`loop.create_server() ` +* The :meth:`loop.create_server() ` method can now accept a list of hosts. (Contributed by Yann Sionneau.) Updates in 3.5.2: -* New :meth:`loop.create_future() ` +* New :meth:`loop.create_future() ` method to create Future objects. This allows alternative event loop implementations, such as `uvloop `_, to provide a faster :class:`asyncio.Future` implementation. (Contributed by Yury Selivanov.) -* New :meth:`loop.get_exception_handler() ` +* New :meth:`loop.get_exception_handler() ` method to get the current exception handler. (Contributed by Yury Selivanov.) @@ -856,13 +856,13 @@ Updates in 3.5.2: sequence appears. (Contributed by Mark Korenberg.) -* The :meth:`loop.create_connection() ` - and :meth:`loop.create_server() ` +* The :meth:`loop.create_connection() ` + and :meth:`loop.create_server() ` methods are optimized to avoid calling the system ``getaddrinfo`` function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.) -* The :meth:`loop.sock_connect(sock, address) ` +* The :meth:`loop.sock_connect(sock, address) ` no longer requires the *address* to be resolved prior to the call. (Contributed by A. Jesse Jiryu Davis.) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 1360af79d073..b413b5a65d52 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -824,7 +824,7 @@ Notable changes in the :mod:`asyncio` module since Python 3.5.0 (Contributed by Yury Selivanov in :issue:`28613`.) * The :func:`~asyncio.ensure_future` function and all functions that - use it, such as :meth:`loop.run_until_complete() `, + use it, such as :meth:`loop.run_until_complete() `, now accept all kinds of :term:`awaitable objects `. (Contributed by Yury Selivanov.) @@ -836,18 +836,18 @@ Notable changes in the :mod:`asyncio` module since Python 3.5.0 method to check if the transport is closing or closed. (Contributed by Yury Selivanov.) -* The :meth:`loop.create_server() ` +* The :meth:`loop.create_server() ` method can now accept a list of hosts. (Contributed by Yann Sionneau.) -* New :meth:`loop.create_future() ` +* New :meth:`loop.create_future() ` method to create Future objects. This allows alternative event loop implementations, such as `uvloop `_, to provide a faster :class:`asyncio.Future` implementation. (Contributed by Yury Selivanov in :issue:`27041`.) -* New :meth:`loop.get_exception_handler() ` +* New :meth:`loop.get_exception_handler() ` method to get the current exception handler. (Contributed by Yury Selivanov in :issue:`27040`.) @@ -860,12 +860,12 @@ Notable changes in the :mod:`asyncio` module since Python 3.5.0 has been improved. (Contributed by Mark Korenberg in :issue:`28370`.) -* The :meth:`loop.getaddrinfo() ` +* The :meth:`loop.getaddrinfo() ` method is optimized to avoid calling the system ``getaddrinfo`` function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.) -* The :meth:`loop.stop() ` +* The :meth:`loop.stop() ` method has been changed to stop the loop immediately after the current iteration. Any new callbacks scheduled as a result of the last iteration will be discarded. @@ -876,7 +876,7 @@ Notable changes in the :mod:`asyncio` module since Python 3.5.0 the :exc:`StopIteration` exception. (Contributed by Chris Angelico in :issue:`26221`.) -* New :meth:`loop.connect_accepted_socket() ` +* New :meth:`loop.connect_accepted_socket() ` method to be used by servers that accept connections outside of asyncio, but that use asyncio to handle them. (Contributed by Jim Fulton in :issue:`27392`.) @@ -884,7 +884,7 @@ Notable changes in the :mod:`asyncio` module since Python 3.5.0 * ``TCP_NODELAY`` flag is now set for all TCP transports by default. (Contributed by Yury Selivanov in :issue:`27456`.) -* New :meth:`loop.shutdown_asyncgens() ` +* New :meth:`loop.shutdown_asyncgens() ` to properly close pending asynchronous generators before closing the loop. (Contributed by Yury Selivanov in :issue:`28003`.) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index cb4865e48403..fbaa2cf15f94 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -637,10 +637,10 @@ include: (Contributed by Yury Selivanov in :issue:`32314`.) * asyncio gained support for :mod:`contextvars`. - :meth:`loop.call_soon() `, - :meth:`loop.call_soon_threadsafe() `, - :meth:`loop.call_later() `, - :meth:`loop.call_at() `, and + :meth:`loop.call_soon() `, + :meth:`loop.call_soon_threadsafe() `, + :meth:`loop.call_later() `, + :meth:`loop.call_at() `, and :meth:`Future.add_done_callback() ` have a new optional keyword-only *context* parameter. :class:`Tasks ` now track their context automatically. @@ -651,11 +651,11 @@ include: to ``asyncio.get_event_loop().create_task()``. (Contributed by Andrew Svetlov in :issue:`32311`.) -* The new :meth:`loop.start_tls() ` +* The new :meth:`loop.start_tls() ` method can be used to upgrade an existing connection to TLS. (Contributed by Yury Selivanov in :issue:`23749`.) -* The new :meth:`loop.sock_recv_into() ` +* The new :meth:`loop.sock_recv_into() ` method allows reading data from a socket directly into a provided buffer making it possible to reduce data copies. (Contributed by Antoine Pitrou in :issue:`31819`.) @@ -683,13 +683,13 @@ include: can be used to determine if the writer is closing. (Contributed by Andrew Svetlov in :issue:`32391`.) -* The new :meth:`loop.sock_sendfile() ` +* The new :meth:`loop.sock_sendfile() ` coroutine method allows sending files using :mod:`os.sendfile` when possible. (Contributed by Andrew Svetlov in :issue:`32410`.) -* The new :meth:`Task.get_loop() ` and - :meth:`Future.get_loop() ` methods - return the instance of the loop on which a task or a future were created. +* The new :meth:`Future.get_loop() ` and + ``Task.get_loop()`` methods return the instance of the loop on which a task or + a future were created. :meth:`Server.get_loop() ` allows doing the same for :class:`asyncio.Server` objects. (Contributed by Yury Selivanov in :issue:`32415` and @@ -698,8 +698,8 @@ include: * It is now possible to control how instances of :class:`asyncio.Server` begin serving. Previously, the server would start serving immediately when created. The new *start_serving* keyword argument to - :meth:`loop.create_server() ` and - :meth:`loop.create_unix_server() `, + :meth:`loop.create_server() ` and + :meth:`loop.create_unix_server() `, as well as :meth:`Server.start_serving() `, and :meth:`Server.serve_forever() ` can be used to decouple server instantiation and serving. The new @@ -717,20 +717,20 @@ include: (Contributed by Yury Selivanov in :issue:`32662`.) * Callback objects returned by - :func:`loop.call_later() ` + :func:`loop.call_later() ` gained the new :meth:`when() ` method which returns an absolute scheduled callback timestamp. (Contributed by Andrew Svetlov in :issue:`32741`.) * The :meth:`loop.create_datagram_endpoint() \ - ` method + ` method gained support for Unix sockets. (Contributed by Quentin Dawans in :issue:`31245`.) * The :func:`asyncio.open_connection`, :func:`asyncio.start_server` functions, - :meth:`loop.create_connection() `, - :meth:`loop.create_server() `, - :meth:`loop.create_accepted_socket() ` + :meth:`loop.create_connection() `, + :meth:`loop.create_server() `, + :meth:`loop.create_accepted_socket() ` methods and their corresponding UNIX socket variants now accept the *ssl_handshake_timeout* keyword argument. (Contributed by Neil Aspinall in :issue:`29970`.) @@ -2360,11 +2360,11 @@ Changes in the Python API (Contributed by Brett Cannon in :issue:`33169`.) * In :mod:`asyncio`, - :meth:`loop.sock_recv() `, - :meth:`loop.sock_sendall() `, - :meth:`loop.sock_accept() `, - :meth:`loop.getaddrinfo() `, - :meth:`loop.getnameinfo() ` + :meth:`loop.sock_recv() `, + :meth:`loop.sock_sendall() `, + :meth:`loop.sock_accept() `, + :meth:`loop.getaddrinfo() `, + :meth:`loop.getnameinfo() ` have been changed to be proper coroutine methods to match their documentation. Previously, these methods returned :class:`asyncio.Future` instances. diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index a9b689d53728..259dcf65f2e0 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -170,7 +170,7 @@ Deprecated * Passing an object that is not an instance of :class:`concurrent.futures.ThreadPoolExecutor` to - :meth:`asyncio.AbstractEventLoop.set_default_executor()` is + :meth:`asyncio.loop.set_default_executor()` is deprecated and will be prohibited in Python 3.9. (Contributed by Elvis Pranskevichus in :issue:`34075`.) @@ -264,7 +264,7 @@ Changes in the Python API * Asyncio tasks can now be named, either by passing the ``name`` keyword argument to :func:`asyncio.create_task` or - the :meth:`~asyncio.AbstractEventLoop.create_task` event loop method, or by + the :meth:`~asyncio.loop.create_task` event loop method, or by calling the :meth:`~asyncio.Task.set_name` method on the task object. The task name is visible in the ``repr()`` output of :class:`asyncio.Task` and can also be retrieved using the :meth:`~asyncio.Task.get_name` method. From webhook-mailer at python.org Tue Sep 11 13:13:08 2018 From: webhook-mailer at python.org (Andrew Svetlov) Date: Tue, 11 Sep 2018 17:13:08 -0000 Subject: [Python-checkins] bpo-34622: Extract asyncio exceptions into a separate module (GH-9141) Message-ID: https://github.com/python/cpython/commit/0baa72f4b2e7185298d09cf64c7b591efcd22af0 commit: 0baa72f4b2e7185298d09cf64c7b591efcd22af0 branch: master author: Andrew Svetlov committer: GitHub date: 2018-09-11T10:13:04-07:00 summary: bpo-34622: Extract asyncio exceptions into a separate module (GH-9141) files: A Lib/asyncio/exceptions.py A Misc/NEWS.d/next/Library/2018-09-10-13-04-40.bpo-34622.tpv_rN.rst M Lib/asyncio/__init__.py M Lib/asyncio/base_events.py M Lib/asyncio/base_futures.py M Lib/asyncio/events.py M Lib/asyncio/futures.py M Lib/asyncio/locks.py M Lib/asyncio/proactor_events.py M Lib/asyncio/streams.py M Lib/asyncio/tasks.py M Lib/asyncio/unix_events.py M Lib/asyncio/windows_events.py M Lib/test/test_asyncio/test_base_events.py M Lib/test/test_asyncio/test_events.py M Lib/test/test_asyncio/test_proactor_events.py M Lib/test/test_asyncio/test_unix_events.py M Modules/_asynciomodule.c diff --git a/Lib/asyncio/__init__.py b/Lib/asyncio/__init__.py index 26859024300e..28c2e2c429f3 100644 --- a/Lib/asyncio/__init__.py +++ b/Lib/asyncio/__init__.py @@ -8,6 +8,7 @@ from .base_events import * from .coroutines import * from .events import * +from .exceptions import * from .futures import * from .locks import * from .protocols import * @@ -25,6 +26,7 @@ __all__ = (base_events.__all__ + coroutines.__all__ + events.__all__ + + exceptions.__all__ + futures.__all__ + locks.__all__ + protocols.__all__ + diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index ee13d1a78028..046743864fdd 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -37,6 +37,7 @@ from . import constants from . import coroutines from . import events +from . import exceptions from . import futures from . import protocols from . import sslproto @@ -327,7 +328,7 @@ def close(self): try: await self._serving_forever_fut - except futures.CancelledError: + except exceptions.CancelledError: try: self.close() await self.wait_closed() @@ -800,7 +801,7 @@ def _getaddrinfo_debug(self, host, port, family, type, proto, flags): try: return await self._sock_sendfile_native(sock, file, offset, count) - except events.SendfileNotAvailableError as exc: + except exceptions.SendfileNotAvailableError as exc: if not fallback: raise return await self._sock_sendfile_fallback(sock, file, @@ -809,7 +810,7 @@ def _getaddrinfo_debug(self, host, port, family, type, proto, flags): async def _sock_sendfile_native(self, sock, file, offset, count): # NB: sendfile syscall is not supported for SSL sockets and # non-mmap files even if sendfile is supported by OS - raise events.SendfileNotAvailableError( + raise exceptions.SendfileNotAvailableError( f"syscall sendfile is not available for socket {sock!r} " "and file {file!r} combination") @@ -1053,7 +1054,7 @@ def _check_sendfile_params(self, sock, file, offset, count): try: return await self._sendfile_native(transport, file, offset, count) - except events.SendfileNotAvailableError as exc: + except exceptions.SendfileNotAvailableError as exc: if not fallback: raise @@ -1066,7 +1067,7 @@ def _check_sendfile_params(self, sock, file, offset, count): offset, count) async def _sendfile_native(self, transp, file, offset, count): - raise events.SendfileNotAvailableError( + raise exceptions.SendfileNotAvailableError( "sendfile syscall is not supported") async def _sendfile_fallback(self, transp, file, offset, count): diff --git a/Lib/asyncio/base_futures.py b/Lib/asyncio/base_futures.py index bd65beec553c..22f298069c50 100644 --- a/Lib/asyncio/base_futures.py +++ b/Lib/asyncio/base_futures.py @@ -1,15 +1,9 @@ __all__ = () -import concurrent.futures import reprlib from . import format_helpers -CancelledError = concurrent.futures.CancelledError -TimeoutError = concurrent.futures.TimeoutError -InvalidStateError = concurrent.futures.InvalidStateError - - # States for Future. _PENDING = 'PENDING' _CANCELLED = 'CANCELLED' diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index 58a60a0b16a0..163b868afeee 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -3,7 +3,7 @@ __all__ = ( 'AbstractEventLoopPolicy', 'AbstractEventLoop', 'AbstractServer', - 'Handle', 'TimerHandle', 'SendfileNotAvailableError', + 'Handle', 'TimerHandle', 'get_event_loop_policy', 'set_event_loop_policy', 'get_event_loop', 'set_event_loop', 'new_event_loop', 'get_child_watcher', 'set_child_watcher', @@ -19,14 +19,7 @@ import threading from . import format_helpers - - -class SendfileNotAvailableError(RuntimeError): - """Sendfile syscall is not available. - - Raised if OS does not support sendfile syscall for given socket or - file type. - """ +from . import exceptions class Handle: diff --git a/Lib/asyncio/exceptions.py b/Lib/asyncio/exceptions.py new file mode 100644 index 000000000000..cac31a54d253 --- /dev/null +++ b/Lib/asyncio/exceptions.py @@ -0,0 +1,60 @@ +"""asyncio exceptions.""" + + +__all__ = ('CancelledError', 'InvalidStateError', 'TimeoutError', + 'IncompleteReadError', 'LimitOverrunError', + 'SendfileNotAvailableError') + +import concurrent.futures +from . import base_futures + + +class CancelledError(concurrent.futures.CancelledError): + """The Future or Task was cancelled.""" + + +class TimeoutError(concurrent.futures.TimeoutError): + """The operation exceeded the given deadline.""" + + +class InvalidStateError(concurrent.futures.InvalidStateError): + """The operation is not allowed in this state.""" + + +class SendfileNotAvailableError(RuntimeError): + """Sendfile syscall is not available. + + Raised if OS does not support sendfile syscall for given socket or + file type. + """ + + +class IncompleteReadError(EOFError): + """ + Incomplete read error. Attributes: + + - partial: read bytes string before the end of stream was reached + - expected: total number of expected bytes (or None if unknown) + """ + def __init__(self, partial, expected): + super().__init__(f'{len(partial)} bytes read on a total of ' + f'{expected!r} expected bytes') + self.partial = partial + self.expected = expected + + def __reduce__(self): + return type(self), (self.partial, self.expected) + + +class LimitOverrunError(Exception): + """Reached the buffer limit while looking for a separator. + + Attributes: + - consumed: total number of to be consumed bytes. + """ + def __init__(self, message, consumed): + super().__init__(message) + self.consumed = consumed + + def __reduce__(self): + return type(self), (self.args[0], self.consumed) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py index 0e0e696a2535..98a5308ed061 100644 --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -1,7 +1,6 @@ """A Future class similar to the one in PEP 3148.""" __all__ = ( - 'CancelledError', 'TimeoutError', 'InvalidStateError', 'Future', 'wrap_future', 'isfuture', ) @@ -12,12 +11,10 @@ from . import base_futures from . import events +from . import exceptions from . import format_helpers -CancelledError = base_futures.CancelledError -InvalidStateError = base_futures.InvalidStateError -TimeoutError = base_futures.TimeoutError isfuture = base_futures.isfuture @@ -170,9 +167,9 @@ def result(self): the future is done and has an exception set, this exception is raised. """ if self._state == _CANCELLED: - raise CancelledError + raise exceptions.CancelledError if self._state != _FINISHED: - raise InvalidStateError('Result is not ready.') + raise exceptions.InvalidStateError('Result is not ready.') self.__log_traceback = False if self._exception is not None: raise self._exception @@ -187,9 +184,9 @@ def exception(self): InvalidStateError. """ if self._state == _CANCELLED: - raise CancelledError + raise exceptions.CancelledError if self._state != _FINISHED: - raise InvalidStateError('Exception is not set.') + raise exceptions.InvalidStateError('Exception is not set.') self.__log_traceback = False return self._exception @@ -231,7 +228,7 @@ def set_result(self, result): InvalidStateError. """ if self._state != _PENDING: - raise InvalidStateError('{}: {!r}'.format(self._state, self)) + raise exceptions.InvalidStateError(f'{self._state}: {self!r}') self._result = result self._state = _FINISHED self.__schedule_callbacks() @@ -243,7 +240,7 @@ def set_exception(self, exception): InvalidStateError. """ if self._state != _PENDING: - raise InvalidStateError('{}: {!r}'.format(self._state, self)) + raise exceptions.InvalidStateError(f'{self._state}: {self!r}') if isinstance(exception, type): exception = exception() if type(exception) is StopIteration: @@ -288,6 +285,18 @@ def _set_result_unless_cancelled(fut, result): fut.set_result(result) +def _convert_future_exc(exc): + exc_class = type(exc) + if exc_class is concurrent.futures.CancelledError: + return exceptions.CancelledError(*exc.args) + elif exc_class is concurrent.futures.TimeoutError: + return exceptions.TimeoutError(*exc.args) + elif exc_class is concurrent.futures.InvalidStateError: + return exceptions.InvalidStateError(*exc.args) + else: + return exc + + def _set_concurrent_future_state(concurrent, source): """Copy state from a future to a concurrent.futures.Future.""" assert source.done() @@ -297,7 +306,7 @@ def _set_concurrent_future_state(concurrent, source): return exception = source.exception() if exception is not None: - concurrent.set_exception(exception) + concurrent.set_exception(_convert_future_exc(exception)) else: result = source.result() concurrent.set_result(result) @@ -317,7 +326,7 @@ def _copy_future_state(source, dest): else: exception = source.exception() if exception is not None: - dest.set_exception(exception) + dest.set_exception(_convert_future_exc(exception)) else: result = source.result() dest.set_result(result) diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py index 91f7a01de8ad..639bd11bd068 100644 --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -7,6 +7,7 @@ from . import events from . import futures +from . import exceptions from .coroutines import coroutine @@ -192,7 +193,7 @@ def locked(self): await fut finally: self._waiters.remove(fut) - except futures.CancelledError: + except exceptions.CancelledError: if not self._locked: self._wake_up_first() raise @@ -363,11 +364,11 @@ def __repr__(self): try: await self.acquire() break - except futures.CancelledError: + except exceptions.CancelledError: cancelled = True if cancelled: - raise futures.CancelledError + raise exceptions.CancelledError async def wait_for(self, predicate): """Wait until a predicate becomes true. diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py index 66bfb0ab11ee..ad23918802fa 100644 --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -15,6 +15,7 @@ from . import constants from . import events from . import futures +from . import exceptions from . import protocols from . import sslproto from . import transports @@ -282,7 +283,7 @@ def _loop_reading(self, fut=None): self._force_close(exc) except OSError as exc: self._fatal_error(exc, 'Fatal read error on pipe transport') - except futures.CancelledError: + except exceptions.CancelledError: if not self._closing: raise else: @@ -555,11 +556,11 @@ def close(self): try: fileno = file.fileno() except (AttributeError, io.UnsupportedOperation) as err: - raise events.SendfileNotAvailableError("not a regular file") + raise exceptions.SendfileNotAvailableError("not a regular file") try: fsize = os.fstat(fileno).st_size except OSError as err: - raise events.SendfileNotAvailableError("not a regular file") + raise exceptions.SendfileNotAvailableError("not a regular file") blocksize = count if count else fsize if not blocksize: return 0 # empty file @@ -615,7 +616,7 @@ def _loop_self_reading(self, f=None): if f is not None: f.result() # may raise f = self._proactor.recv(self._ssock, 4096) - except futures.CancelledError: + except exceptions.CancelledError: # _close_self_pipe() has been called, stop waiting for data return except Exception as exc: @@ -666,7 +667,7 @@ def loop(f=None): elif self._debug: logger.debug("Accept failed on socket %r", sock, exc_info=True) - except futures.CancelledError: + except exceptions.CancelledError: sock.close() else: self._accept_futures[sock.fileno()] = f diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py index d6531f88a74d..9dab49b35e46 100644 --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -1,8 +1,6 @@ __all__ = ( 'StreamReader', 'StreamWriter', 'StreamReaderProtocol', - 'open_connection', 'start_server', - 'IncompleteReadError', 'LimitOverrunError', -) + 'open_connection', 'start_server') import socket @@ -11,6 +9,7 @@ from . import coroutines from . import events +from . import exceptions from . import protocols from .log import logger from .tasks import sleep @@ -19,37 +18,6 @@ _DEFAULT_LIMIT = 2 ** 16 # 64 KiB -class IncompleteReadError(EOFError): - """ - Incomplete read error. Attributes: - - - partial: read bytes string before the end of stream was reached - - expected: total number of expected bytes (or None if unknown) - """ - def __init__(self, partial, expected): - super().__init__(f'{len(partial)} bytes read on a total of ' - f'{expected!r} expected bytes') - self.partial = partial - self.expected = expected - - def __reduce__(self): - return type(self), (self.partial, self.expected) - - -class LimitOverrunError(Exception): - """Reached the buffer limit while looking for a separator. - - Attributes: - - consumed: total number of to be consumed bytes. - """ - def __init__(self, message, consumed): - super().__init__(message) - self.consumed = consumed - - def __reduce__(self): - return type(self), (self.args[0], self.consumed) - - async def open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds): """A wrapper for create_connection() returning a (reader, writer) pair. @@ -494,9 +462,9 @@ def feed_data(self, data): seplen = len(sep) try: line = await self.readuntil(sep) - except IncompleteReadError as e: + except exceptions.IncompleteReadError as e: return e.partial - except LimitOverrunError as e: + except exceptions.LimitOverrunError as e: if self._buffer.startswith(sep, e.consumed): del self._buffer[:e.consumed + seplen] else: @@ -571,7 +539,7 @@ def feed_data(self, data): # see upper comment for explanation. offset = buflen + 1 - seplen if offset > self._limit: - raise LimitOverrunError( + raise exceptions.LimitOverrunError( 'Separator is not found, and chunk exceed the limit', offset) @@ -582,13 +550,13 @@ def feed_data(self, data): if self._eof: chunk = bytes(self._buffer) self._buffer.clear() - raise IncompleteReadError(chunk, None) + raise exceptions.IncompleteReadError(chunk, None) # _wait_for_data() will resume reading if stream was paused. await self._wait_for_data('readuntil') if isep > self._limit: - raise LimitOverrunError( + raise exceptions.LimitOverrunError( 'Separator is found, but chunk is longer than limit', isep) chunk = self._buffer[:isep + seplen] @@ -674,7 +642,7 @@ def feed_data(self, data): if self._eof: incomplete = bytes(self._buffer) self._buffer.clear() - raise IncompleteReadError(incomplete, n) + raise exceptions.IncompleteReadError(incomplete, n) await self._wait_for_data('readexactly') diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 03d71d37f01a..7121aa65da0e 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -21,6 +21,7 @@ from . import base_tasks from . import coroutines from . import events +from . import exceptions from . import futures from .coroutines import coroutine @@ -228,11 +229,11 @@ def cancel(self): def __step(self, exc=None): if self.done(): - raise futures.InvalidStateError( + raise exceptions.InvalidStateError( f'_step(): already done: {self!r}, {exc!r}') if self._must_cancel: - if not isinstance(exc, futures.CancelledError): - exc = futures.CancelledError() + if not isinstance(exc, exceptions.CancelledError): + exc = exceptions.CancelledError() self._must_cancel = False coro = self._coro self._fut_waiter = None @@ -250,10 +251,10 @@ def __step(self, exc=None): if self._must_cancel: # Task is cancelled right before coro stops. self._must_cancel = False - super().set_exception(futures.CancelledError()) + super().set_exception(exceptions.CancelledError()) else: super().set_result(exc.value) - except futures.CancelledError: + except exceptions.CancelledError: super().cancel() # I.e., Future.cancel(self). except Exception as exc: super().set_exception(exc) @@ -419,7 +420,7 @@ def _release_waiter(waiter, *args): return fut.result() fut.cancel() - raise futures.TimeoutError() + raise exceptions.TimeoutError() waiter = loop.create_future() timeout_handle = loop.call_later(timeout, _release_waiter, waiter) @@ -432,7 +433,7 @@ def _release_waiter(waiter, *args): # wait until the future completes or the timeout try: await waiter - except futures.CancelledError: + except exceptions.CancelledError: fut.remove_done_callback(cb) fut.cancel() raise @@ -445,7 +446,7 @@ def _release_waiter(waiter, *args): # after wait_for() returns. # See https://bugs.python.org/issue32751 await _cancel_and_wait(fut, loop=loop) - raise futures.TimeoutError() + raise exceptions.TimeoutError() finally: timeout_handle.cancel() @@ -554,7 +555,7 @@ def _on_completion(f): f = await done.get() if f is None: # Dummy value from _on_timeout(). - raise futures.TimeoutError + raise exceptions.TimeoutError return f.result() # May raise f.exception(). for f in todo: @@ -701,7 +702,7 @@ def _done_callback(fut): # Check if 'fut' is cancelled first, as # 'fut.exception()' will *raise* a CancelledError # instead of returning it. - exc = futures.CancelledError() + exc = exceptions.CancelledError() outer.set_exception(exc) return else: @@ -720,7 +721,7 @@ def _done_callback(fut): # Check if 'fut' is cancelled first, as # 'fut.exception()' will *raise* a CancelledError # instead of returning it. - res = futures.CancelledError() + res = exceptions.CancelledError() else: res = fut.exception() if res is None: @@ -731,7 +732,7 @@ def _done_callback(fut): # If gather is being cancelled we must propagate the # cancellation regardless of *return_exceptions* argument. # See issue 32684. - outer.set_exception(futures.CancelledError()) + outer.set_exception(exceptions.CancelledError()) else: outer.set_result(results) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index 7cad7e3637a1..1a62db4f59bc 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -18,6 +18,7 @@ from . import constants from . import coroutines from . import events +from . import exceptions from . import futures from . import selector_events from . import tasks @@ -319,16 +320,16 @@ def _child_watcher_callback(self, pid, returncode, transp): try: os.sendfile except AttributeError as exc: - raise events.SendfileNotAvailableError( + raise exceptions.SendfileNotAvailableError( "os.sendfile() is not available") try: fileno = file.fileno() except (AttributeError, io.UnsupportedOperation) as err: - raise events.SendfileNotAvailableError("not a regular file") + raise exceptions.SendfileNotAvailableError("not a regular file") try: fsize = os.fstat(fileno).st_size except OSError as err: - raise events.SendfileNotAvailableError("not a regular file") + raise exceptions.SendfileNotAvailableError("not a regular file") blocksize = count if count else fsize if not blocksize: return 0 # empty file @@ -382,7 +383,7 @@ def _sock_sendfile_native_impl(self, fut, registered_fd, sock, fileno, # one being 'file' is not a regular mmap(2)-like # file, in which case we'll fall back on using # plain send(). - err = events.SendfileNotAvailableError( + err = exceptions.SendfileNotAvailableError( "os.sendfile call failed") self._sock_sendfile_update_filepos(fileno, offset, total_sent) fut.set_exception(err) diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py index 2ec542764375..fdde8e9e0bf4 100644 --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -12,6 +12,7 @@ from . import events from . import base_subprocess from . import futures +from . import exceptions from . import proactor_events from . import selector_events from . import tasks @@ -351,7 +352,7 @@ def loop_accept_pipe(f=None): elif self._debug: logger.warning("Accept pipe failed on pipe %r", pipe, exc_info=True) - except futures.CancelledError: + except exceptions.CancelledError: if pipe: pipe.close() else: @@ -497,7 +498,7 @@ def finish_accept(trans, key, ov): # Coroutine closing the accept socket if the future is cancelled try: await future - except futures.CancelledError: + except exceptions.CancelledError: conn.close() raise diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py index e10863795335..fe3c38371d0c 100644 --- a/Lib/test/test_asyncio/test_base_events.py +++ b/Lib/test/test_asyncio/test_base_events.py @@ -1946,7 +1946,7 @@ def cleanup(): def test__sock_sendfile_native_failure(self): sock, proto = self.prepare() - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "sendfile is not available"): self.run_loop(self.loop._sock_sendfile_native(sock, self.file, 0, None)) @@ -1957,7 +1957,7 @@ def test__sock_sendfile_native_failure(self): def test_sock_sendfile_no_fallback(self): sock, proto = self.prepare() - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "sendfile is not available"): self.run_loop(self.loop.sock_sendfile(sock, self.file, fallback=False)) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py index 11cd950df1ce..01f616ab5b19 100644 --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -2393,7 +2393,7 @@ def sendfile_native(transp, file, offset, count): self.loop._sendfile_native = sendfile_native - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not supported"): self.run_loop( self.loop.sendfile(cli_proto.transport, self.file, diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py index dddbe3b2a107..afc4c19a96b0 100644 --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -951,7 +951,7 @@ def cleanup(): def test_sock_sendfile_not_a_file(self): sock, proto = self.prepare() f = object() - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not a regular file"): self.run_loop(self.loop._sock_sendfile_native(sock, f, 0, None)) @@ -960,7 +960,7 @@ def test_sock_sendfile_not_a_file(self): def test_sock_sendfile_iobuffer(self): sock, proto = self.prepare() f = io.BytesIO() - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not a regular file"): self.run_loop(self.loop._sock_sendfile_native(sock, f, 0, None)) @@ -970,7 +970,7 @@ def test_sock_sendfile_not_regular_file(self): sock, proto = self.prepare() f = mock.Mock() f.fileno.return_value = -1 - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not a regular file"): self.run_loop(self.loop._sock_sendfile_native(sock, f, 0, None)) diff --git a/Lib/test/test_asyncio/test_unix_events.py b/Lib/test/test_asyncio/test_unix_events.py index 29b345b142e8..62545c0f98c3 100644 --- a/Lib/test/test_asyncio/test_unix_events.py +++ b/Lib/test/test_asyncio/test_unix_events.py @@ -521,7 +521,7 @@ def cleanup(): def test_sock_sendfile_not_available(self): sock, proto = self.prepare() with mock.patch('asyncio.unix_events.os', spec=[]): - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "os[.]sendfile[(][)] is not available"): self.run_loop(self.loop._sock_sendfile_native(sock, self.file, 0, None)) @@ -530,7 +530,7 @@ def test_sock_sendfile_not_available(self): def test_sock_sendfile_not_a_file(self): sock, proto = self.prepare() f = object() - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not a regular file"): self.run_loop(self.loop._sock_sendfile_native(sock, f, 0, None)) @@ -539,7 +539,7 @@ def test_sock_sendfile_not_a_file(self): def test_sock_sendfile_iobuffer(self): sock, proto = self.prepare() f = io.BytesIO() - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not a regular file"): self.run_loop(self.loop._sock_sendfile_native(sock, f, 0, None)) @@ -549,7 +549,7 @@ def test_sock_sendfile_not_regular_file(self): sock, proto = self.prepare() f = mock.Mock() f.fileno.return_value = -1 - with self.assertRaisesRegex(events.SendfileNotAvailableError, + with self.assertRaisesRegex(asyncio.SendfileNotAvailableError, "not a regular file"): self.run_loop(self.loop._sock_sendfile_native(sock, f, 0, None)) @@ -605,7 +605,7 @@ def test_sock_sendfile_os_error_first_call(self): with self.assertRaises(KeyError): self.loop._selector.get_key(sock) exc = fut.exception() - self.assertIsInstance(exc, events.SendfileNotAvailableError) + self.assertIsInstance(exc, asyncio.SendfileNotAvailableError) self.assertEqual(0, self.file.tell()) def test_sock_sendfile_os_error_next_call(self): @@ -630,7 +630,7 @@ def test_sock_sendfile_exception(self): fileno = self.file.fileno() fut = self.loop.create_future() - err = events.SendfileNotAvailableError() + err = asyncio.SendfileNotAvailableError() with mock.patch('os.sendfile', side_effect=err): self.loop._sock_sendfile_native_impl(fut, sock.fileno(), sock, fileno, diff --git a/Misc/NEWS.d/next/Library/2018-09-10-13-04-40.bpo-34622.tpv_rN.rst b/Misc/NEWS.d/next/Library/2018-09-10-13-04-40.bpo-34622.tpv_rN.rst new file mode 100644 index 000000000000..493d6abd910b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-09-10-13-04-40.bpo-34622.tpv_rN.rst @@ -0,0 +1,4 @@ +Create a dedicated ``asyncio.CancelledError``, ``asyncio.InvalidStateError`` +and ``asyncio.TimeoutError`` exception classes. Inherit them from +corresponding exceptions from ``concurrent.futures`` package. Extract +``asyncio`` exceptions into a separate file. diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index 3d7ce01a680c..2ed033cacd85 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -3306,6 +3306,8 @@ module_init(void) WITH_MOD("asyncio.base_futures") GET_MOD_ATTR(asyncio_future_repr_info_func, "_future_repr_info") + + WITH_MOD("asyncio.exceptions") GET_MOD_ATTR(asyncio_InvalidStateError, "InvalidStateError") GET_MOD_ATTR(asyncio_CancelledError, "CancelledError") From webhook-mailer at python.org Tue Sep 11 13:29:52 2018 From: webhook-mailer at python.org (Berker Peksag) Date: Tue, 11 Sep 2018 17:29:52 -0000 Subject: [Python-checkins] bpo-29386: Pass -1 to epoll_wait() when timeout is < -1 (GH-9040) Message-ID: https://github.com/python/cpython/commit/b690b9b04729ba3d91c59bff1bb11c3dcc1b50fc commit: b690b9b04729ba3d91c59bff1bb11c3dcc1b50fc branch: master author: Berker Peksag committer: GitHub date: 2018-09-11T20:29:48+03:00 summary: bpo-29386: Pass -1 to epoll_wait() when timeout is < -1 (GH-9040) Although the kernel accepts any negative value for timeout, the documented value to block indefinitely is -1. This commit also makes the code similar to select.poll.poll(). files: M Modules/selectmodule.c diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index 836af5429af5..d86727a8978d 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -1498,17 +1498,12 @@ select_epoll_poll_impl(pyEpoll_Object *self, PyObject *timeout_obj, int nfds, i; PyObject *elist = NULL, *etuple = NULL; struct epoll_event *evs = NULL; - _PyTime_t timeout, ms, deadline; + _PyTime_t timeout = -1, ms = -1, deadline = 0; if (self->epfd < 0) return pyepoll_err_closed(); - if (timeout_obj == Py_None) { - timeout = -1; - ms = -1; - deadline = 0; /* initialize to prevent gcc warning */ - } - else { + if (timeout_obj != Py_None) { /* epoll_wait() has a resolution of 1 millisecond, round towards infinity to wait at least timeout seconds. */ if (_PyTime_FromSecondsObject(&timeout, timeout_obj, @@ -1525,8 +1520,20 @@ select_epoll_poll_impl(pyEpoll_Object *self, PyObject *timeout_obj, PyErr_SetString(PyExc_OverflowError, "timeout is too large"); return NULL; } + /* epoll_wait(2) treats all arbitrary negative numbers the same + for the timeout argument, but -1 is the documented way to block + indefinitely in the epoll_wait(2) documentation, so we set ms + to -1 if the value of ms is a negative number. + + Note that we didn't use INFTIM here since it's non-standard and + isn't available under Linux. */ + if (ms < 0) { + ms = -1; + } - deadline = _PyTime_GetMonotonicClock() + timeout; + if (timeout >= 0) { + deadline = _PyTime_GetMonotonicClock() + timeout; + } } if (maxevents == -1) { From webhook-mailer at python.org Tue Sep 11 13:35:12 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 17:35:12 -0000 Subject: [Python-checkins] Fix Tools/gdb/libpython.py Message-ID: https://github.com/python/cpython/commit/ed74a258aab2d5ca56a311acffdc7a46ca8ce2a1 commit: ed74a258aab2d5ca56a311acffdc7a46ca8ce2a1 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T10:35:08-07:00 summary: Fix Tools/gdb/libpython.py Backport https://github.com/python/cpython/commit/11659d00b9185c8f02ea6b642fa475a80e21f1a9 into this change instead of leaving it a separate followup change. files: A Misc/NEWS.d/next/Core and Builtins/2018-04-19-08-30-07.bpo-33312.mDe2iL.rst M Objects/dict-common.h M Objects/dictobject.c M Tools/gdb/libpython.py diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-04-19-08-30-07.bpo-33312.mDe2iL.rst b/Misc/NEWS.d/next/Core and Builtins/2018-04-19-08-30-07.bpo-33312.mDe2iL.rst new file mode 100644 index 000000000000..40b51b902c9f --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-04-19-08-30-07.bpo-33312.mDe2iL.rst @@ -0,0 +1,3 @@ +Fixed clang ubsan (undefined behavior sanitizer) warnings in dictobject.c by +adjusting how the internal struct _dictkeysobject shared keys structure is +declared. diff --git a/Objects/dict-common.h b/Objects/dict-common.h index ce9edabd8959..d3ae07bfd3f4 100644 --- a/Objects/dict-common.h +++ b/Objects/dict-common.h @@ -59,15 +59,8 @@ struct _dictkeysobject { - 4 bytes if dk_size <= 0xffffffff (int32_t*) - 8 bytes otherwise (int64_t*) - Dynamically sized, 8 is minimum. */ - union { - int8_t as_1[8]; - int16_t as_2[4]; - int32_t as_4[2]; -#if SIZEOF_VOID_P > 4 - int64_t as_8[1]; -#endif - } dk_indices; + Dynamically sized, SIZEOF_VOID_P is minimum. */ + char dk_indices[]; /* char is required to avoid strict aliasing. */ /* "PyDictKeyEntry dk_entries[dk_usable];" array follows: see the DK_ENTRIES() macro */ diff --git a/Objects/dictobject.c b/Objects/dictobject.c index ddd05826ea5f..0768d1154542 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -300,7 +300,7 @@ PyDict_Fini(void) 2 : sizeof(int32_t)) #endif #define DK_ENTRIES(dk) \ - ((PyDictKeyEntry*)(&(dk)->dk_indices.as_1[DK_SIZE(dk) * DK_IXSIZE(dk)])) + ((PyDictKeyEntry*)(&((int8_t*)((dk)->dk_indices))[DK_SIZE(dk) * DK_IXSIZE(dk)])) #define DK_DEBUG_INCREF _Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA #define DK_DEBUG_DECREF _Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA @@ -318,21 +318,21 @@ dk_get_index(PyDictKeysObject *keys, Py_ssize_t i) Py_ssize_t ix; if (s <= 0xff) { - int8_t *indices = keys->dk_indices.as_1; + int8_t *indices = (int8_t*)(keys->dk_indices); ix = indices[i]; } else if (s <= 0xffff) { - int16_t *indices = keys->dk_indices.as_2; + int16_t *indices = (int16_t*)(keys->dk_indices); ix = indices[i]; } #if SIZEOF_VOID_P > 4 else if (s > 0xffffffff) { - int64_t *indices = keys->dk_indices.as_8; + int64_t *indices = (int64_t*)(keys->dk_indices); ix = indices[i]; } #endif else { - int32_t *indices = keys->dk_indices.as_4; + int32_t *indices = (int32_t*)(keys->dk_indices); ix = indices[i]; } assert(ix >= DKIX_DUMMY); @@ -348,23 +348,23 @@ dk_set_index(PyDictKeysObject *keys, Py_ssize_t i, Py_ssize_t ix) assert(ix >= DKIX_DUMMY); if (s <= 0xff) { - int8_t *indices = keys->dk_indices.as_1; + int8_t *indices = (int8_t*)(keys->dk_indices); assert(ix <= 0x7f); indices[i] = (char)ix; } else if (s <= 0xffff) { - int16_t *indices = keys->dk_indices.as_2; + int16_t *indices = (int16_t*)(keys->dk_indices); assert(ix <= 0x7fff); indices[i] = (int16_t)ix; } #if SIZEOF_VOID_P > 4 else if (s > 0xffffffff) { - int64_t *indices = keys->dk_indices.as_8; + int64_t *indices = (int64_t*)(keys->dk_indices); indices[i] = ix; } #endif else { - int32_t *indices = keys->dk_indices.as_4; + int32_t *indices = (int32_t*)(keys->dk_indices); assert(ix <= 0x7fffffff); indices[i] = (int32_t)ix; } @@ -424,8 +424,8 @@ static PyDictKeysObject empty_keys_struct = { lookdict_split, /* dk_lookup */ 0, /* dk_usable (immutable) */ 0, /* dk_nentries */ - .dk_indices = { .as_1 = {DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, - DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY}}, + {DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, + DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY}, /* dk_indices */ }; static PyObject *empty_values[1] = { NULL }; @@ -533,7 +533,6 @@ static PyDictKeysObject *new_keys_object(Py_ssize_t size) } else { dk = PyObject_MALLOC(sizeof(PyDictKeysObject) - - Py_MEMBER_SIZE(PyDictKeysObject, dk_indices) + es * size + sizeof(PyDictKeyEntry) * usable); if (dk == NULL) { @@ -546,7 +545,7 @@ static PyDictKeysObject *new_keys_object(Py_ssize_t size) dk->dk_usable = usable; dk->dk_lookup = lookdict_unicode_nodummy; dk->dk_nentries = 0; - memset(&dk->dk_indices.as_1[0], 0xff, es * size); + memset(&dk->dk_indices[0], 0xff, es * size); memset(DK_ENTRIES(dk), 0, sizeof(PyDictKeyEntry) * usable); return dk; } @@ -3086,7 +3085,6 @@ _PyDict_SizeOf(PyDictObject *mp) in the type object. */ if (mp->ma_keys->dk_refcnt == 1) res += (sizeof(PyDictKeysObject) - - Py_MEMBER_SIZE(PyDictKeysObject, dk_indices) + DK_IXSIZE(mp->ma_keys) * size + sizeof(PyDictKeyEntry) * usable); return res; @@ -3096,7 +3094,6 @@ Py_ssize_t _PyDict_KeysSize(PyDictKeysObject *keys) { return (sizeof(PyDictKeysObject) - - Py_MEMBER_SIZE(PyDictKeysObject, dk_indices) + DK_IXSIZE(keys) * DK_SIZE(keys) + USABLE_FRACTION(DK_SIZE(keys)) * sizeof(PyDictKeyEntry)); } diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 6d7a94247bc3..6d3d52f397d4 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -736,7 +736,7 @@ def _get_entries(self, keys): else: offset = 8 * dk_size - ent_addr = keys['dk_indices']['as_1'].address + ent_addr = keys['dk_indices'].address ent_addr = ent_addr.cast(_type_unsigned_char_ptr()) + offset ent_ptr_t = gdb.lookup_type('PyDictKeyEntry').pointer() ent_addr = ent_addr.cast(ent_ptr_t) From webhook-mailer at python.org Tue Sep 11 13:44:56 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 17:44:56 -0000 Subject: [Python-checkins] bpo-28617 Fixed docs inaccuracies about the types that support membership tests (GH-9086) Message-ID: https://github.com/python/cpython/commit/08bcf647d8a92e4bd47531588b284c6820b7a7ef commit: 08bcf647d8a92e4bd47531588b284c6820b7a7ef branch: master author: wim glenn committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> date: 2018-09-11T10:44:52-07:00 summary: bpo-28617 Fixed docs inaccuracies about the types that support membership tests (GH-9086) https://bugs.python.org/issue28617 files: A Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 87c64214c67d..f7f59cd3a6bb 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -197,8 +197,8 @@ exception. operator: not in Two more operations with the same syntactic priority, :keyword:`in` and -:keyword:`not in`, are supported only by sequence types (below). - +:keyword:`not in`, are supported by types that are :term:`iterable` or +implement the :meth:`__contains__` method. .. _typesnumeric: diff --git a/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst b/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst new file mode 100644 index 000000000000..281afad71d27 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst @@ -0,0 +1,2 @@ +Fixed info in the stdtypes docs concerning the types that support membership +tests. From webhook-mailer at python.org Tue Sep 11 13:47:49 2018 From: webhook-mailer at python.org (Steve Dower) Date: Tue, 11 Sep 2018 17:47:49 -0000 Subject: [Python-checkins] Update VSTS to Azure DevOps and simplify dependencies (GH-9168) Message-ID: https://github.com/python/cpython/commit/b7d613fc5636d5d8c5ec32d8004c095e20cc4d7d commit: b7d613fc5636d5d8c5ec32d8004c095e20cc4d7d branch: master author: Steve Dower committer: GitHub date: 2018-09-11T10:47:46-07:00 summary: Update VSTS to Azure DevOps and simplify dependencies (GH-9168) files: A .vsts/install_deps.sh D .vsts/linux-deps.yml M .vsts/docs.yml M .vsts/linux-buildbot.yml M .vsts/linux-coverage.yml M .vsts/linux-pr.yml M .vsts/macos-buildbot.yml M .vsts/macos-pr.yml M README.rst diff --git a/.vsts/docs.yml b/.vsts/docs.yml index 93a7282f770a..0be07b31dfcc 100644 --- a/.vsts/docs.yml +++ b/.vsts/docs.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: diff --git a/.vsts/install_deps.sh b/.vsts/install_deps.sh new file mode 100755 index 000000000000..7b98cfddb6c4 --- /dev/null +++ b/.vsts/install_deps.sh @@ -0,0 +1,19 @@ +sudo apt-get update + +sudo apt-get -yq install \ + build-essential \ + zlib1g-dev \ + libbz2-dev \ + liblzma-dev \ + libncurses5-dev \ + libreadline6-dev \ + libsqlite3-dev \ + libssl-dev \ + libgdbm-dev \ + tk-dev \ + lzma \ + lzma-dev \ + liblzma-dev \ + libffi-dev \ + uuid-dev \ + xvfb diff --git a/.vsts/linux-buildbot.yml b/.vsts/linux-buildbot.yml index fc2c8ca2486e..517040048937 100644 --- a/.vsts/linux-buildbot.yml +++ b/.vsts/linux-buildbot.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -30,31 +30,9 @@ steps: #- template: linux-deps.yml -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - - script: echo ##vso[task.prependpath]$(OPENSSL_DIR) displayName: 'Add $(OPENSSL_DIR) to PATH' -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' - script: python3 Tools/ssl/multissltests.py --steps=library --base-directory $(build.sourcesDirectory)/multissl --openssl $(OPENSSL) --system Linux displayName: 'python multissltests.py' diff --git a/.vsts/linux-coverage.yml b/.vsts/linux-coverage.yml index 1112555ab93f..cc03e4258ab4 100644 --- a/.vsts/linux-coverage.yml +++ b/.vsts/linux-coverage.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -40,32 +40,9 @@ steps: #- template: linux-deps.yml -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - - script: echo ##vso[task.prependpath]$(OPENSSL_DIR) displayName: 'Add $(OPENSSL_DIR) to PATH' -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - script: python3 Tools/ssl/multissltests.py --steps=library --base-directory $(build.sourcesDirectory)/multissl --openssl $(OPENSSL) --system Linux diff --git a/.vsts/linux-deps.yml b/.vsts/linux-deps.yml deleted file mode 100644 index 83b0b5961721..000000000000 --- a/.vsts/linux-deps.yml +++ /dev/null @@ -1,37 +0,0 @@ -# Note: this file is not currently used, but when template support comes to VSTS it -# will be referenced from the other scripts.. - -# Current docs for the syntax of this file are at: -# https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted.md - -parameters: - OPENSSL: 1.1.0g - OPENSSL_DIR: "$(build.sourcesDirectory)/multissl/openssl/$(OPENSSL)" - -steps: -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - -- script: echo ##vso[task.prependpath]$(OPENSSL_DIR) - displayName: 'Add $(OPENSSL_DIR) to PATH' -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb - displayName: 'Install dependencies' -- script: python3 Tools/ssl/multissltests.py --steps=library --base-directory $(build.sourcesDirectory)/multissl --openssl $(OPENSSL) --system Linux - displayName: 'python multissltests.py' diff --git a/.vsts/linux-pr.yml b/.vsts/linux-pr.yml index 145ebb38016a..6e4ac7c65c4d 100644 --- a/.vsts/linux-pr.yml +++ b/.vsts/linux-pr.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -40,34 +40,11 @@ steps: #- template: linux-deps.yml -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - - script: echo ##vso[task.prependpath]$(OPENSSL_DIR) displayName: 'Add $(OPENSSL_DIR) to PATH' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) diff --git a/.vsts/macos-buildbot.yml b/.vsts/macos-buildbot.yml index d9b2297283b0..f58ea1626144 100644 --- a/.vsts/macos-buildbot.yml +++ b/.vsts/macos-buildbot.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted macOS Preview + name: Hosted macOS trigger: branches: diff --git a/.vsts/macos-pr.yml b/.vsts/macos-pr.yml index a3fd4487ed3a..c56e66b5090b 100644 --- a/.vsts/macos-pr.yml +++ b/.vsts/macos-pr.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted macOS Preview + name: Hosted macOS trigger: branches: diff --git a/README.rst b/README.rst index 6047e7aef2cb..c65f6e29d413 100644 --- a/README.rst +++ b/README.rst @@ -9,17 +9,17 @@ This is Python version 3.8.0 alpha 0 :alt: CPython build status on Appveyor :target: https://ci.appveyor.com/project/python/cpython/branch/master -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/Linux-Buildbot?branchName=master&label=Linux - :alt: CPython build status on VSTS (Linux) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=6&branchName=master +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Linux-Buildbot?branchName=master&label=Linux + :alt: CPython build status on Azure DevOps (Linux) + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=6&branchName=master -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/macOS-Buildbot?branchName=master&label=macOS - :alt: CPython build status on VSTS (macOS) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=5&branchName=master +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/macOS-Buildbot?branchName=master&label=macOS + :alt: CPython build status on Azure DevOps (macOS) + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=5&branchName=master -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/Windows-Buildbot?branchName=master&label=Windows - :alt: CPython build status on VSTS (Windows) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=4&branchName=master +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Windows-Buildbot?branchName=master&label=Windows + :alt: CPython build status on Azure DevOps (Windows) + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=master .. image:: https://codecov.io/gh/python/cpython/branch/master/graph/badge.svg :alt: CPython code coverage on Codecov From webhook-mailer at python.org Tue Sep 11 14:13:38 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 18:13:38 -0000 Subject: [Python-checkins] bpo-28617 Fixed docs inaccuracies about the types that support membership tests (GH-9086) Message-ID: https://github.com/python/cpython/commit/3e648f8371e342ccfa663ad77e82a538fcc8c9fb commit: 3e648f8371e342ccfa663ad77e82a538fcc8c9fb branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T11:13:33-07:00 summary: bpo-28617 Fixed docs inaccuracies about the types that support membership tests (GH-9086) https://bugs.python.org/issue28617 (cherry picked from commit 08bcf647d8a92e4bd47531588b284c6820b7a7ef) Co-authored-by: wim glenn files: A Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index e71ce75b8329..efa0a89490e1 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -197,8 +197,8 @@ exception. operator: not in Two more operations with the same syntactic priority, :keyword:`in` and -:keyword:`not in`, are supported only by sequence types (below). - +:keyword:`not in`, are supported by types that are :term:`iterable` or +implement the :meth:`__contains__` method. .. _typesnumeric: diff --git a/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst b/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst new file mode 100644 index 000000000000..281afad71d27 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst @@ -0,0 +1,2 @@ +Fixed info in the stdtypes docs concerning the types that support membership +tests. From webhook-mailer at python.org Tue Sep 11 14:18:19 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 18:18:19 -0000 Subject: [Python-checkins] bpo-28617 Fixed docs inaccuracies about the types that support membership tests (GH-9086) Message-ID: https://github.com/python/cpython/commit/889f080a4d5cdb1cfb901b953f4b89f3ea806bbe commit: 889f080a4d5cdb1cfb901b953f4b89f3ea806bbe branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T11:18:15-07:00 summary: bpo-28617 Fixed docs inaccuracies about the types that support membership tests (GH-9086) https://bugs.python.org/issue28617 (cherry picked from commit 08bcf647d8a92e4bd47531588b284c6820b7a7ef) Co-authored-by: wim glenn files: A Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 0172d34ade6c..6de7e1a2df5a 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -197,8 +197,8 @@ exception. operator: not in Two more operations with the same syntactic priority, :keyword:`in` and -:keyword:`not in`, are supported only by sequence types (below). - +:keyword:`not in`, are supported by types that are :term:`iterable` or +implement the :meth:`__contains__` method. .. _typesnumeric: diff --git a/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst b/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst new file mode 100644 index 000000000000..281afad71d27 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-09-06-22-39-47.bpo-28617.MjnJLz.rst @@ -0,0 +1,2 @@ +Fixed info in the stdtypes docs concerning the types that support membership +tests. From webhook-mailer at python.org Tue Sep 11 14:45:31 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 18:45:31 -0000 Subject: [Python-checkins] bpo-34613: document the correct value of limit argument of asyncio.StreamReader (GH-9121) Message-ID: https://github.com/python/cpython/commit/b4ec36200a959da70eba94c19826446a8efdffdd commit: b4ec36200a959da70eba94c19826446a8efdffdd branch: master author: Bram committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> date: 2018-09-11T11:45:26-07:00 summary: bpo-34613: document the correct value of limit argument of asyncio.StreamReader (GH-9121) The default value of asyncio.StreamReader *limit* is `_DEFAULT_LIMIT` instead of `None`. https://bugs.python.org/issue34613 files: M Doc/library/asyncio-stream.rst diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index 3fe7ac7b4abc..27b5205f1c5b 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -146,10 +146,12 @@ and work with streams: StreamReader ============ -.. class:: StreamReader(limit=None, loop=None) +.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) This class is :ref:`not thread safe `. + The *limit* argument's default value is set to _DEFAULT_LIMIT which is 2**16 (64 KiB) + .. method:: exception() Get the exception. From webhook-mailer at python.org Tue Sep 11 14:46:58 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 18:46:58 -0000 Subject: [Python-checkins] closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-3788) Message-ID: https://github.com/python/cpython/commit/24bd50bdcc97d65130c07d6cd26085fd06c3e972 commit: 24bd50bdcc97d65130c07d6cd26085fd06c3e972 branch: master author: Oren Milman committer: Benjamin Peterson date: 2018-09-11T11:46:55-07:00 summary: closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-3788) files: A Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst M Lib/test/test_deque.py M Modules/_collectionsmodule.c diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index e895c3c9f0df..921136069d77 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -892,6 +892,21 @@ def __iter__(self): d1 == d2 # not clear if this is supposed to be True or False, # but it used to give a SystemError + @support.cpython_only + def test_bug_31608(self): + # The interpreter used to crash in specific cases where a deque + # subclass returned a non-deque. + class X(deque): + pass + d = X() + def bad___new__(cls, *args, **kwargs): + return [42] + X.__new__ = bad___new__ + with self.assertRaises(TypeError): + d * 42 # shouldn't crash + with self.assertRaises(TypeError): + d + deque([1, 2, 3]) # shouldn't crash + class SubclassWithKwargs(deque): def __init__(self, newarg=1): diff --git a/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst new file mode 100644 index 000000000000..d657a8697361 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst @@ -0,0 +1,2 @@ +Raise a ``TypeError`` instead of crashing if a ``collections.deque`` subclass +returns a non-deque from ``__new__``. Patch by Oren Milman. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index c04485771402..935b4348a8ff 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -478,6 +478,7 @@ deque_inplace_concat(dequeobject *deque, PyObject *other) static PyObject * deque_copy(PyObject *deque, PyObject *Py_UNUSED(ignored)) { + PyObject *result; dequeobject *old_deque = (dequeobject *)deque; if (Py_TYPE(deque) == &deque_type) { dequeobject *new_deque; @@ -502,11 +503,19 @@ deque_copy(PyObject *deque, PyObject *Py_UNUSED(ignored)) return NULL; } if (old_deque->maxlen < 0) - return PyObject_CallFunctionObjArgs((PyObject *)(Py_TYPE(deque)), - deque, NULL); + result = PyObject_CallFunctionObjArgs((PyObject *)(Py_TYPE(deque)), + deque, NULL); else - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", - deque, old_deque->maxlen, NULL); + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", + deque, old_deque->maxlen, NULL); + if (result != NULL && !PyObject_TypeCheck(result, &deque_type)) { + PyErr_Format(PyExc_TypeError, + "%.200s() must return a deque, not %.200s", + Py_TYPE(deque)->tp_name, Py_TYPE(result)->tp_name); + Py_DECREF(result); + return NULL; + } + return result; } PyDoc_STRVAR(copy_doc, "Return a shallow copy of a deque."); From webhook-mailer at python.org Tue Sep 11 14:57:51 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 18:57:51 -0000 Subject: [Python-checkins] bpo-34613: document the correct value of limit argument of asyncio.StreamReader (GH-9121) Message-ID: https://github.com/python/cpython/commit/cb51dd7cac6a6e2a7ba67fa4cd328a68f630095b commit: cb51dd7cac6a6e2a7ba67fa4cd328a68f630095b branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T11:57:45-07:00 summary: bpo-34613: document the correct value of limit argument of asyncio.StreamReader (GH-9121) The default value of asyncio.StreamReader *limit* is `_DEFAULT_LIMIT` instead of `None`. https://bugs.python.org/issue34613 (cherry picked from commit b4ec36200a959da70eba94c19826446a8efdffdd) Co-authored-by: Bram files: M Doc/library/asyncio-stream.rst diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index 491afdd610ca..a510e1e638a7 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -88,10 +88,12 @@ Stream functions StreamReader ============ -.. class:: StreamReader(limit=None, loop=None) +.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) This class is :ref:`not thread safe `. + The *limit* argument's default value is set to _DEFAULT_LIMIT which is 2**16 (64 KiB) + .. method:: exception() Get the exception. From webhook-mailer at python.org Tue Sep 11 14:59:33 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 18:59:33 -0000 Subject: [Python-checkins] bpo-34613: document the correct value of limit argument of asyncio.StreamReader (GH-9121) Message-ID: https://github.com/python/cpython/commit/e02ca4270ef258162215e345c23025bec27f9eb0 commit: e02ca4270ef258162215e345c23025bec27f9eb0 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T11:59:29-07:00 summary: bpo-34613: document the correct value of limit argument of asyncio.StreamReader (GH-9121) The default value of asyncio.StreamReader *limit* is `_DEFAULT_LIMIT` instead of `None`. https://bugs.python.org/issue34613 (cherry picked from commit b4ec36200a959da70eba94c19826446a8efdffdd) Co-authored-by: Bram files: M Doc/library/asyncio-stream.rst diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index f662e7223337..ca7daabdadd5 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -126,10 +126,12 @@ Stream functions StreamReader ============ -.. class:: StreamReader(limit=None, loop=None) +.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) This class is :ref:`not thread safe `. + The *limit* argument's default value is set to _DEFAULT_LIMIT which is 2**16 (64 KiB) + .. method:: exception() Get the exception. From webhook-mailer at python.org Tue Sep 11 15:08:15 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 19:08:15 -0000 Subject: [Python-checkins] closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-3788) Message-ID: https://github.com/python/cpython/commit/536e45accf8f05355dd943a6966b9968cdb15f5a commit: 536e45accf8f05355dd943a6966b9968cdb15f5a branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T12:08:10-07:00 summary: closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-3788) (cherry picked from commit 24bd50bdcc97d65130c07d6cd26085fd06c3e972) Co-authored-by: Oren Milman files: A Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst M Lib/test/test_deque.py M Modules/_collectionsmodule.c diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index e895c3c9f0df..921136069d77 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -892,6 +892,21 @@ def __iter__(self): d1 == d2 # not clear if this is supposed to be True or False, # but it used to give a SystemError + @support.cpython_only + def test_bug_31608(self): + # The interpreter used to crash in specific cases where a deque + # subclass returned a non-deque. + class X(deque): + pass + d = X() + def bad___new__(cls, *args, **kwargs): + return [42] + X.__new__ = bad___new__ + with self.assertRaises(TypeError): + d * 42 # shouldn't crash + with self.assertRaises(TypeError): + d + deque([1, 2, 3]) # shouldn't crash + class SubclassWithKwargs(deque): def __init__(self, newarg=1): diff --git a/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst new file mode 100644 index 000000000000..d657a8697361 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst @@ -0,0 +1,2 @@ +Raise a ``TypeError`` instead of crashing if a ``collections.deque`` subclass +returns a non-deque from ``__new__``. Patch by Oren Milman. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 4116d4053547..cfd8905edeb3 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -514,6 +514,7 @@ deque_inplace_concat(dequeobject *deque, PyObject *other) static PyObject * deque_copy(PyObject *deque) { + PyObject *result; dequeobject *old_deque = (dequeobject *)deque; if (Py_TYPE(deque) == &deque_type) { dequeobject *new_deque; @@ -538,11 +539,19 @@ deque_copy(PyObject *deque) return NULL; } if (old_deque->maxlen < 0) - return PyObject_CallFunctionObjArgs((PyObject *)(Py_TYPE(deque)), - deque, NULL); + result = PyObject_CallFunctionObjArgs((PyObject *)(Py_TYPE(deque)), + deque, NULL); else - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", - deque, old_deque->maxlen, NULL); + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", + deque, old_deque->maxlen, NULL); + if (result != NULL && !PyObject_TypeCheck(result, &deque_type)) { + PyErr_Format(PyExc_TypeError, + "%.200s() must return a deque, not %.200s", + Py_TYPE(deque)->tp_name, Py_TYPE(result)->tp_name); + Py_DECREF(result); + return NULL; + } + return result; } PyDoc_STRVAR(copy_doc, "Return a shallow copy of a deque."); From webhook-mailer at python.org Tue Sep 11 15:12:46 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 19:12:46 -0000 Subject: [Python-checkins] [3.6] closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-9178) Message-ID: https://github.com/python/cpython/commit/ccbbdd0a1e00ecad6f0005438dd6ff6d84fd9ceb commit: ccbbdd0a1e00ecad6f0005438dd6ff6d84fd9ceb branch: 3.6 author: Benjamin Peterson committer: GitHub date: 2018-09-11T12:12:42-07:00 summary: [3.6] closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-9178) (cherry picked from commit 24bd50bdcc97d65130c07d6cd26085fd06c3e972) Co-authored-by: Oren Milman files: A Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst M Lib/test/test_deque.py M Modules/_collectionsmodule.c diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index ce517b51d58e..ed9ea0251a74 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -893,6 +893,21 @@ def __iter__(self): d1 == d2 # not clear if this is supposed to be True or False, # but it used to give a SystemError + @support.cpython_only + def test_bug_31608(self): + # The interpreter used to crash in specific cases where a deque + # subclass returned a non-deque. + class X(deque): + pass + d = X() + def bad___new__(cls, *args, **kwargs): + return [42] + X.__new__ = bad___new__ + with self.assertRaises(TypeError): + d * 42 # shouldn't crash + with self.assertRaises(TypeError): + d + deque([1, 2, 3]) # shouldn't crash + class SubclassWithKwargs(deque): def __init__(self, newarg=1): diff --git a/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst new file mode 100644 index 000000000000..d657a8697361 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst @@ -0,0 +1,2 @@ +Raise a ``TypeError`` instead of crashing if a ``collections.deque`` subclass +returns a non-deque from ``__new__``. Patch by Oren Milman. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index d7b344be692c..85037d002703 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -514,6 +514,7 @@ deque_inplace_concat(dequeobject *deque, PyObject *other) static PyObject * deque_copy(PyObject *deque) { + PyObject *result; dequeobject *old_deque = (dequeobject *)deque; if (Py_TYPE(deque) == &deque_type) { dequeobject *new_deque; @@ -538,10 +539,19 @@ deque_copy(PyObject *deque) return NULL; } if (old_deque->maxlen < 0) - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "O", deque, NULL); + result = PyObject_CallFunctionObjArgs((PyObject *)(Py_TYPE(deque)), + deque, NULL); else - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", - deque, old_deque->maxlen, NULL); + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", + deque, old_deque->maxlen, NULL); + if (result != NULL && !PyObject_TypeCheck(result, &deque_type)) { + PyErr_Format(PyExc_TypeError, + "%.200s() must return a deque, not %.200s", + Py_TYPE(deque)->tp_name, Py_TYPE(result)->tp_name); + Py_DECREF(result); + return NULL; + } + return result; } PyDoc_STRVAR(copy_doc, "Return a shallow copy of a deque."); From webhook-mailer at python.org Tue Sep 11 16:42:01 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 20:42:01 -0000 Subject: [Python-checkins] [2.7] closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-9179) Message-ID: https://github.com/python/cpython/commit/253279c616d4f34287c5749df15e20eb2eb988d6 commit: 253279c616d4f34287c5749df15e20eb2eb988d6 branch: 2.7 author: Benjamin Peterson committer: GitHub date: 2018-09-11T13:41:57-07:00 summary: [2.7] closes bpo-31608: Fix a crash in methods of a subclass of _collections.deque with a bad __new__(). (GH-9179) files: A Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst M Lib/test/test_deque.py M Modules/_collectionsmodule.c diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index c81064d9f2a6..e6307ab5c32e 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -659,6 +659,21 @@ def __iter__(self): d1 == d2 # not clear if this is supposed to be True or False, # but it used to give a SystemError + @test_support.cpython_only + def test_bug_31608(self): + # The interpreter used to crash in specific cases where a deque + # subclass returned a non-deque. + class X(deque): + pass + d = X() + def bad___new__(cls, *args, **kwargs): + return [42] + X.__new__ = bad___new__ + with self.assertRaises(TypeError): + d * 42 # shouldn't crash + with self.assertRaises(TypeError): + d + deque([1, 2, 3]) # shouldn't crash + class SubclassWithKwargs(deque): def __init__(self, newarg=1): diff --git a/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst new file mode 100644 index 000000000000..d657a8697361 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2017-10-29-10-37-55.bpo-31608.wkp8Nw.rst @@ -0,0 +1,2 @@ +Raise a ``TypeError`` instead of crashing if a ``collections.deque`` subclass +returns a non-deque from ``__new__``. Patch by Oren Milman. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 9364aba549b1..3ca393082f1d 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -859,11 +859,20 @@ deque_traverse(dequeobject *deque, visitproc visit, void *arg) static PyObject * deque_copy(PyObject *deque) { + PyObject *result; if (((dequeobject *)deque)->maxlen == -1) - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "O", deque, NULL); + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "O", deque, NULL); else - return PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", + result = PyObject_CallFunction((PyObject *)(Py_TYPE(deque)), "Oi", deque, ((dequeobject *)deque)->maxlen, NULL); + if (result != NULL && !PyObject_TypeCheck(result, &deque_type)) { + PyErr_Format(PyExc_TypeError, + "%.200s() must return a deque, not %.200s", + Py_TYPE(deque)->tp_name, Py_TYPE(result)->tp_name); + Py_DECREF(result); + return NULL; + } + return result; } PyDoc_STRVAR(copy_doc, "Return a shallow copy of a deque."); From webhook-mailer at python.org Tue Sep 11 16:55:01 2018 From: webhook-mailer at python.org (Benjamin Peterson) Date: Tue, 11 Sep 2018 20:55:01 -0000 Subject: [Python-checkins] [2.7] bpo-32502: Discard 64-bit (and other invalid) hardware addresses (GH-9125) Message-ID: https://github.com/python/cpython/commit/d919c60e6936f853ad15040017f2c0bce0f027f8 commit: d919c60e6936f853ad15040017f2c0bce0f027f8 branch: 2.7 author: Chih-Hsuan Yen committer: Benjamin Peterson date: 2018-09-11T13:54:57-07:00 summary: [2.7] bpo-32502: Discard 64-bit (and other invalid) hardware addresses (GH-9125) (cherry picked from commit 6b273f7f4056f8276f61a97c789d6bb4425e653c) Co-authored-by: Bo Bayles files: A Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst M Lib/test/test_uuid.py M Lib/uuid.py diff --git a/Lib/test/test_uuid.py b/Lib/test/test_uuid.py index 3749564c930e..6cd2c392932e 100644 --- a/Lib/test/test_uuid.py +++ b/Lib/test/test_uuid.py @@ -287,6 +287,39 @@ def test_getnode(self): node2 = uuid.getnode() self.assertEqual(node1, node2, '%012x != %012x' % (node1, node2)) + # bpo-32502: UUID1 requires a 48-bit identifier, but hardware identifiers + # need not necessarily be 48 bits (e.g., EUI-64). + def test_uuid1_eui64(self): + # Confirm that uuid.getnode ignores hardware addresses larger than 48 + # bits. Mock out each platform's *_getnode helper functions to return + # something just larger than 48 bits to test. This will cause + # uuid.getnode to fall back on uuid._random_getnode, which will + # generate a valid value. + too_large_getter = lambda: 1 << 48 + + uuid_real__node = uuid._node + uuid_real__NODE_GETTERS_WIN32 = uuid._NODE_GETTERS_WIN32 + uuid_real__NODE_GETTERS_UNIX = uuid._NODE_GETTERS_UNIX + uuid._node = None + uuid._NODE_GETTERS_WIN32 = [too_large_getter] + uuid._NODE_GETTERS_UNIX = [too_large_getter] + try: + node = uuid.getnode() + finally: + uuid._node = uuid_real__node + uuid._NODE_GETTERS_WIN32 = uuid_real__NODE_GETTERS_WIN32 + uuid._NODE_GETTERS_UNIX = uuid_real__NODE_GETTERS_UNIX + + self.assertTrue(0 < node < (1 << 48), '%012x' % node) + + # Confirm that uuid1 can use the generated node, i.e., the that + # uuid.getnode fell back on uuid._random_getnode() rather than using + # the value from too_large_getter above. + try: + uuid.uuid1(node=node) + except ValueError as e: + self.fail('uuid1 was given an invalid node ID') + @unittest.skipUnless(importable('ctypes'), 'requires ctypes') def test_uuid1(self): equal = self.assertEqual diff --git a/Lib/uuid.py b/Lib/uuid.py index 973013c00636..80d33c0bd83f 100644 --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -522,6 +522,11 @@ def _random_getnode(): _node = None +_NODE_GETTERS_WIN32 = [_windll_getnode, _netbios_getnode, _ipconfig_getnode] + +_NODE_GETTERS_UNIX = [_unixdll_getnode, _ifconfig_getnode, _arp_getnode, + _lanscan_getnode, _netstat_getnode] + def getnode(): """Get the hardware address as a 48-bit positive integer. @@ -537,18 +542,19 @@ def getnode(): import sys if sys.platform == 'win32': - getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode] + getters = _NODE_GETTERS_WIN32 else: - getters = [_unixdll_getnode, _ifconfig_getnode, _arp_getnode, - _lanscan_getnode, _netstat_getnode] + getters = _NODE_GETTERS_UNIX for getter in getters + [_random_getnode]: try: _node = getter() except: continue - if _node is not None: + if (_node is not None) and (0 <= _node < (1 << 48)): return _node + assert False, '_random_getnode() returned invalid value: {}'.format(_node) + _last_timestamp = None diff --git a/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst b/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst new file mode 100644 index 000000000000..8338632aa2c5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-01-20-17-15-34.bpo-32502.OXJfn7.rst @@ -0,0 +1,2 @@ +uuid.uuid1 no longer raises an exception if a 64-bit hardware address is +encountered. From webhook-mailer at python.org Tue Sep 11 16:59:28 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 20:59:28 -0000 Subject: [Python-checkins] bpo-34365: Update date object documentation (GH-8814) Message-ID: https://github.com/python/cpython/commit/9c223794c754408644c16349b85dd27fdba8a926 commit: 9c223794c754408644c16349b85dd27fdba8a926 branch: master author: Danish Prakash committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> date: 2018-09-11T13:59:23-07:00 summary: bpo-34365: Update date object documentation (GH-8814) Python 3.x does not fall back to comparing object addresses when comparing two `dt` objects. https://bugs.python.org/issue34365 files: M Doc/library/datetime.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 53c181c4014a..22582476cf95 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -526,10 +526,9 @@ Notes: (4) In other words, ``date1 < date2`` if and only if ``date1.toordinal() < - date2.toordinal()``. In order to stop comparison from falling back to the - default scheme of comparing object addresses, date comparison normally raises - :exc:`TypeError` if the other comparand isn't also a :class:`date` object. - However, ``NotImplemented`` is returned instead if the other comparand has a + date2.toordinal()``. Date comparison raises :exc:`TypeError` if + the other comparand isn't also a :class:`date` object. However, + ``NotImplemented`` is returned instead if the other comparand has a :meth:`timetuple` attribute. This hook gives other kinds of date objects a chance at implementing mixed-type comparison. If not, when a :class:`date` object is compared to an object of a different type, :exc:`TypeError` is raised From webhook-mailer at python.org Tue Sep 11 17:01:17 2018 From: webhook-mailer at python.org (Steve Dower) Date: Tue, 11 Sep 2018 21:01:17 -0000 Subject: [Python-checkins] Update VSTS to Azure DevOps and simplify dependencies (GH-9168) Message-ID: https://github.com/python/cpython/commit/d289df13d376d69764884982c236dbc3879ad43c commit: d289df13d376d69764884982c236dbc3879ad43c branch: 3.7 author: Steve Dower committer: GitHub date: 2018-09-11T14:01:13-07:00 summary: Update VSTS to Azure DevOps and simplify dependencies (GH-9168) files: A .vsts/install_deps.sh D .vsts/linux-deps.yml M .vsts/docs.yml M .vsts/linux-buildbot.yml M .vsts/linux-coverage.yml M .vsts/linux-pr.yml M .vsts/macos-buildbot.yml M .vsts/macos-pr.yml M README.rst diff --git a/.vsts/docs.yml b/.vsts/docs.yml index 93a7282f770a..0be07b31dfcc 100644 --- a/.vsts/docs.yml +++ b/.vsts/docs.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: diff --git a/.vsts/install_deps.sh b/.vsts/install_deps.sh new file mode 100755 index 000000000000..7b98cfddb6c4 --- /dev/null +++ b/.vsts/install_deps.sh @@ -0,0 +1,19 @@ +sudo apt-get update + +sudo apt-get -yq install \ + build-essential \ + zlib1g-dev \ + libbz2-dev \ + liblzma-dev \ + libncurses5-dev \ + libreadline6-dev \ + libsqlite3-dev \ + libssl-dev \ + libgdbm-dev \ + tk-dev \ + lzma \ + lzma-dev \ + liblzma-dev \ + libffi-dev \ + uuid-dev \ + xvfb diff --git a/.vsts/linux-buildbot.yml b/.vsts/linux-buildbot.yml index fc2c8ca2486e..517040048937 100644 --- a/.vsts/linux-buildbot.yml +++ b/.vsts/linux-buildbot.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -30,31 +30,9 @@ steps: #- template: linux-deps.yml -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - - script: echo ##vso[task.prependpath]$(OPENSSL_DIR) displayName: 'Add $(OPENSSL_DIR) to PATH' -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' - script: python3 Tools/ssl/multissltests.py --steps=library --base-directory $(build.sourcesDirectory)/multissl --openssl $(OPENSSL) --system Linux displayName: 'python multissltests.py' diff --git a/.vsts/linux-coverage.yml b/.vsts/linux-coverage.yml index 1112555ab93f..cc03e4258ab4 100644 --- a/.vsts/linux-coverage.yml +++ b/.vsts/linux-coverage.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -40,32 +40,9 @@ steps: #- template: linux-deps.yml -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - - script: echo ##vso[task.prependpath]$(OPENSSL_DIR) displayName: 'Add $(OPENSSL_DIR) to PATH' -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - script: python3 Tools/ssl/multissltests.py --steps=library --base-directory $(build.sourcesDirectory)/multissl --openssl $(OPENSSL) --system Linux diff --git a/.vsts/linux-deps.yml b/.vsts/linux-deps.yml deleted file mode 100644 index 83b0b5961721..000000000000 --- a/.vsts/linux-deps.yml +++ /dev/null @@ -1,37 +0,0 @@ -# Note: this file is not currently used, but when template support comes to VSTS it -# will be referenced from the other scripts.. - -# Current docs for the syntax of this file are at: -# https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted.md - -parameters: - OPENSSL: 1.1.0g - OPENSSL_DIR: "$(build.sourcesDirectory)/multissl/openssl/$(OPENSSL)" - -steps: -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - -- script: echo ##vso[task.prependpath]$(OPENSSL_DIR) - displayName: 'Add $(OPENSSL_DIR) to PATH' -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb - displayName: 'Install dependencies' -- script: python3 Tools/ssl/multissltests.py --steps=library --base-directory $(build.sourcesDirectory)/multissl --openssl $(OPENSSL) --system Linux - displayName: 'python multissltests.py' diff --git a/.vsts/linux-pr.yml b/.vsts/linux-pr.yml index 145ebb38016a..6e4ac7c65c4d 100644 --- a/.vsts/linux-pr.yml +++ b/.vsts/linux-pr.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -40,34 +40,11 @@ steps: #- template: linux-deps.yml -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - - script: echo ##vso[task.prependpath]$(OPENSSL_DIR) displayName: 'Add $(OPENSSL_DIR) to PATH' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) diff --git a/.vsts/macos-buildbot.yml b/.vsts/macos-buildbot.yml index d9b2297283b0..f58ea1626144 100644 --- a/.vsts/macos-buildbot.yml +++ b/.vsts/macos-buildbot.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted macOS Preview + name: Hosted macOS trigger: branches: diff --git a/.vsts/macos-pr.yml b/.vsts/macos-pr.yml index a3fd4487ed3a..c56e66b5090b 100644 --- a/.vsts/macos-pr.yml +++ b/.vsts/macos-pr.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted macOS Preview + name: Hosted macOS trigger: branches: diff --git a/README.rst b/README.rst index b330b41abdb3..377aa3dfce6f 100644 --- a/README.rst +++ b/README.rst @@ -9,17 +9,17 @@ This is Python version 3.7.0+ :alt: CPython build status on Appveyor :target: https://ci.appveyor.com/project/python/cpython/branch/master -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/Linux-Buildbot?branchName=3.7&label=Linux +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Linux-Buildbot?branchName=3.7&label=Linux :alt: CPython build status on VSTS (Linux) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=6&branchName=3.7 + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=6&branchName=3.7 -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/macOS-Buildbot?branchName=3.7&label=macOS +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/macOS-Buildbot?branchName=3.7&label=macOS :alt: CPython build status on VSTS (macOS) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=5&branchName=3.7 + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=5&branchName=3.7 -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/Windows-Buildbot?branchName=3.7&label=Windows +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Windows-Buildbot?branchName=3.7&label=Windows :alt: CPython build status on VSTS (Windows) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=4&branchName=3.7 + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=3.7 .. image:: https://codecov.io/gh/python/cpython/branch/master/graph/badge.svg :alt: CPython code coverage on Codecov From webhook-mailer at python.org Tue Sep 11 17:01:22 2018 From: webhook-mailer at python.org (Steve Dower) Date: Tue, 11 Sep 2018 21:01:22 -0000 Subject: [Python-checkins] Update VSTS to Azure DevOps and simplify dependencies (GH-9168) Message-ID: https://github.com/python/cpython/commit/7f76eebf407c5caf8564ec3244ed7409b828974d commit: 7f76eebf407c5caf8564ec3244ed7409b828974d branch: 3.6 author: Steve Dower committer: GitHub date: 2018-09-11T14:01:19-07:00 summary: Update VSTS to Azure DevOps and simplify dependencies (GH-9168) files: A .vsts/install_deps.sh D .vsts/linux-deps.yml M .vsts/docs.yml M .vsts/linux-buildbot.yml M .vsts/linux-coverage.yml M .vsts/linux-pr.yml M .vsts/macos-buildbot.yml M .vsts/macos-pr.yml M README.rst diff --git a/.vsts/docs.yml b/.vsts/docs.yml index 93a7282f770a..0be07b31dfcc 100644 --- a/.vsts/docs.yml +++ b/.vsts/docs.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: diff --git a/.vsts/install_deps.sh b/.vsts/install_deps.sh new file mode 100755 index 000000000000..7b98cfddb6c4 --- /dev/null +++ b/.vsts/install_deps.sh @@ -0,0 +1,19 @@ +sudo apt-get update + +sudo apt-get -yq install \ + build-essential \ + zlib1g-dev \ + libbz2-dev \ + liblzma-dev \ + libncurses5-dev \ + libreadline6-dev \ + libsqlite3-dev \ + libssl-dev \ + libgdbm-dev \ + tk-dev \ + lzma \ + lzma-dev \ + liblzma-dev \ + libffi-dev \ + uuid-dev \ + xvfb diff --git a/.vsts/linux-buildbot.yml b/.vsts/linux-buildbot.yml index 4c01bdd165d1..09084e07df77 100644 --- a/.vsts/linux-buildbot.yml +++ b/.vsts/linux-buildbot.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -25,31 +25,7 @@ steps: clean: true fetchDepth: 5 -#- template: linux-deps.yml - -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' - script: ./configure --with-pydebug diff --git a/.vsts/linux-coverage.yml b/.vsts/linux-coverage.yml index 62fafe347688..4eb75efc8040 100644 --- a/.vsts/linux-coverage.yml +++ b/.vsts/linux-coverage.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -35,34 +35,9 @@ steps: displayName: Detect doc-only changes condition: and(succeeded(), variables['system.pullRequest.targetBranch']) -#- template: linux-deps.yml - -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' - + condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - script: ./configure --with-pydebug displayName: 'Configure CPython (debug)' diff --git a/.vsts/linux-deps.yml b/.vsts/linux-deps.yml deleted file mode 100644 index d1922a7c3237..000000000000 --- a/.vsts/linux-deps.yml +++ /dev/null @@ -1,31 +0,0 @@ -# Note: this file is not currently used, but when template support comes to VSTS it -# will be referenced from the other scripts.. - -# Current docs for the syntax of this file are at: -# https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted.md - -#parameters: - -steps: -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb - displayName: 'Install dependencies' diff --git a/.vsts/linux-pr.yml b/.vsts/linux-pr.yml index 922fa81bc414..945ac097e5e8 100644 --- a/.vsts/linux-pr.yml +++ b/.vsts/linux-pr.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted Linux Preview + name: 'Hosted Ubuntu 1604' trigger: branches: @@ -35,36 +35,10 @@ steps: displayName: Detect doc-only changes condition: and(succeeded(), variables['system.pullRequest.targetBranch']) -#- template: linux-deps.yml - -# See https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md -# For now, we copy/paste the steps -- script: echo "deb-src http://archive.ubuntu.com/ubuntu/ xenial main" > /etc/apt/sources.list.d/python.list && sudo apt-get update - displayName: 'Update apt-get lists' - condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - -- script: > - sudo apt-get -yq install - build-essential - zlib1g-dev - libbz2-dev - liblzma-dev - libncurses5-dev - libreadline6-dev - libsqlite3-dev - libssl-dev - libgdbm-dev - tk-dev - lzma - lzma-dev - liblzma-dev - libffi-dev - uuid-dev - xvfb +- script: ./.vsts/install_deps.sh displayName: 'Install dependencies' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) - - script: ./configure --with-pydebug displayName: 'Configure CPython (debug)' condition: and(succeeded(), ne(variables['DocOnly'], 'true')) diff --git a/.vsts/macos-buildbot.yml b/.vsts/macos-buildbot.yml index d9b2297283b0..f58ea1626144 100644 --- a/.vsts/macos-buildbot.yml +++ b/.vsts/macos-buildbot.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted macOS Preview + name: Hosted macOS trigger: branches: diff --git a/.vsts/macos-pr.yml b/.vsts/macos-pr.yml index a3fd4487ed3a..c56e66b5090b 100644 --- a/.vsts/macos-pr.yml +++ b/.vsts/macos-pr.yml @@ -4,7 +4,7 @@ name: $(BuildDefinitionName)_$(Date:yyyyMMdd)$(Rev:.rr) queue: - name: Hosted macOS Preview + name: Hosted macOS trigger: branches: diff --git a/README.rst b/README.rst index c9b518bcf2cb..a2c76ac71dee 100644 --- a/README.rst +++ b/README.rst @@ -9,17 +9,17 @@ This is Python version 3.6.6+ :alt: CPython build status on Appveyor :target: https://ci.appveyor.com/project/python/cpython/branch/3.6 -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/Linux-Buildbot?branchName=3.6&label=Linux +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Linux-Buildbot?branchName=3.6&label=Linux :alt: CPython build status on VSTS (Linux) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=6&branchName=3.6 + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=6&branchName=3.6 -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/macOS-Buildbot?branchName=3.6&label=macOS +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/macOS-Buildbot?branchName=3.6&label=macOS :alt: CPython build status on VSTS (macOS) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=5&branchName=3.6 + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=5&branchName=3.6 -.. image:: https://python.visualstudio.com/cpython/_apis/build/status/Windows-Buildbot?branchName=3.6&label=Windows +.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Windows-Buildbot?branchName=3.6&label=Windows :alt: CPython build status on VSTS (Windows) - :target: https://python.visualstudio.com/cpython/_build/latest?definitionId=4&branchName=3.6 + :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=3.6 .. image:: https://codecov.io/gh/python/cpython/branch/3.6/graph/badge.svg :alt: CPython code coverage on Codecov From webhook-mailer at python.org Tue Sep 11 17:24:56 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 21:24:56 -0000 Subject: [Python-checkins] bpo-34365: Update date object documentation (GH-8814) Message-ID: https://github.com/python/cpython/commit/e2b40f4ce954ea3d35a73541029b2253abd9d245 commit: e2b40f4ce954ea3d35a73541029b2253abd9d245 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T14:24:53-07:00 summary: bpo-34365: Update date object documentation (GH-8814) Python 3.x does not fall back to comparing object addresses when comparing two `dt` objects. https://bugs.python.org/issue34365 (cherry picked from commit 9c223794c754408644c16349b85dd27fdba8a926) Co-authored-by: Danish Prakash files: M Doc/library/datetime.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 53c181c4014a..22582476cf95 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -526,10 +526,9 @@ Notes: (4) In other words, ``date1 < date2`` if and only if ``date1.toordinal() < - date2.toordinal()``. In order to stop comparison from falling back to the - default scheme of comparing object addresses, date comparison normally raises - :exc:`TypeError` if the other comparand isn't also a :class:`date` object. - However, ``NotImplemented`` is returned instead if the other comparand has a + date2.toordinal()``. Date comparison raises :exc:`TypeError` if + the other comparand isn't also a :class:`date` object. However, + ``NotImplemented`` is returned instead if the other comparand has a :meth:`timetuple` attribute. This hook gives other kinds of date objects a chance at implementing mixed-type comparison. If not, when a :class:`date` object is compared to an object of a different type, :exc:`TypeError` is raised From webhook-mailer at python.org Tue Sep 11 17:27:09 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 21:27:09 -0000 Subject: [Python-checkins] bpo-34365: Update date object documentation (GH-8814) Message-ID: https://github.com/python/cpython/commit/8a2c2d4e1b6b46a9a982e5c064ca4399ec28d55d commit: 8a2c2d4e1b6b46a9a982e5c064ca4399ec28d55d branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2018-09-11T14:27:06-07:00 summary: bpo-34365: Update date object documentation (GH-8814) Python 3.x does not fall back to comparing object addresses when comparing two `dt` objects. https://bugs.python.org/issue34365 (cherry picked from commit 9c223794c754408644c16349b85dd27fdba8a926) Co-authored-by: Danish Prakash files: M Doc/library/datetime.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 89f85fc11ad4..88bc328e3a1d 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -510,10 +510,9 @@ Notes: (4) In other words, ``date1 < date2`` if and only if ``date1.toordinal() < - date2.toordinal()``. In order to stop comparison from falling back to the - default scheme of comparing object addresses, date comparison normally raises - :exc:`TypeError` if the other comparand isn't also a :class:`date` object. - However, ``NotImplemented`` is returned instead if the other comparand has a + date2.toordinal()``. Date comparison raises :exc:`TypeError` if + the other comparand isn't also a :class:`date` object. However, + ``NotImplemented`` is returned instead if the other comparand has a :meth:`timetuple` attribute. This hook gives other kinds of date objects a chance at implementing mixed-type comparison. If not, when a :class:`date` object is compared to an object of a different type, :exc:`TypeError` is raised From webhook-mailer at python.org Tue Sep 11 17:45:49 2018 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Sep 2018 21:45:49 -0000 Subject: [Python-checkins] closes bpo-31902: Fix the col_offset attribute for ast.Async* nodes to point to the "async" keyword. (GH-4175) Message-ID: https://github.com/python/cpython/commit/90fc8980bbcc5c7dcced3627fe172b0bfd193a3b commit: 90fc8980bbcc5c7dcced3627fe172b0bfd193a3b branch: master author: guoci committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> date: 2018-09-11T14:45:45-07:00 summary: closes bpo-31902: Fix the col_offset attribute for ast.Async* nodes to point to the "async" keyword. (GH-4175) Previously, col_offset points to the keyword after "async". files: A Misc/NEWS.d/next/Core and Builtins/2017-10-30-12-44-50.bpo-31902.a07fa57.rst M Lib/test/test_ast.py M Python/ast.c diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 7db40e7797f7..72f8467847e8 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -1181,12 +1181,12 @@ def main(): ('Module', [('Expr', (1, 0), ('DictComp', (1, 0), ('Name', (1, 1), 'a', ('Load',)), ('Name', (1, 5), 'b', ('Load',)), [('comprehension', ('Tuple', (1, 11), [('Name', (1, 11), 'v', ('Store',)), ('Name', (1, 13), 'w', ('Store',))], ('Store',)), ('Name', (1, 18), 'x', ('Load',)), [], 0)]))]), ('Module', [('Expr', (1, 0), ('SetComp', (1, 0), ('Name', (1, 1), 'r', ('Load',)), [('comprehension', ('Name', (1, 7), 'l', ('Store',)), ('Name', (1, 12), 'x', ('Load',)), [('Name', (1, 17), 'g', ('Load',))], 0)]))]), ('Module', [('Expr', (1, 0), ('SetComp', (1, 0), ('Name', (1, 1), 'r', ('Load',)), [('comprehension', ('Tuple', (1, 7), [('Name', (1, 7), 'l', ('Store',)), ('Name', (1, 9), 'm', ('Store',))], ('Store',)), ('Name', (1, 14), 'x', ('Load',)), [], 0)]))]), -('Module', [('AsyncFunctionDef', (1, 6), 'f', ('arguments', [], None, [], [], None, []), [('Expr', (2, 1), ('Str', (2, 1), 'async function')), ('Expr', (3, 1), ('Await', (3, 1), ('Call', (3, 7), ('Name', (3, 7), 'something', ('Load',)), [], [])))], [], None)]), -('Module', [('AsyncFunctionDef', (1, 6), 'f', ('arguments', [], None, [], [], None, []), [('AsyncFor', (2, 7), ('Name', (2, 11), 'e', ('Store',)), ('Name', (2, 16), 'i', ('Load',)), [('Expr', (2, 19), ('Num', (2, 19), 1))], [('Expr', (3, 7), ('Num', (3, 7), 2))])], [], None)]), -('Module', [('AsyncFunctionDef', (1, 6), 'f', ('arguments', [], None, [], [], None, []), [('AsyncWith', (2, 7), [('withitem', ('Name', (2, 12), 'a', ('Load',)), ('Name', (2, 17), 'b', ('Store',)))], [('Expr', (2, 20), ('Num', (2, 20), 1))])], [], None)]), +('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Expr', (2, 1), ('Str', (2, 1), 'async function')), ('Expr', (3, 1), ('Await', (3, 1), ('Call', (3, 7), ('Name', (3, 7), 'something', ('Load',)), [], [])))], [], None)]), +('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('AsyncFor', (2, 1), ('Name', (2, 11), 'e', ('Store',)), ('Name', (2, 16), 'i', ('Load',)), [('Expr', (2, 19), ('Num', (2, 19), 1))], [('Expr', (3, 7), ('Num', (3, 7), 2))])], [], None)]), +('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('AsyncWith', (2, 1), [('withitem', ('Name', (2, 12), 'a', ('Load',)), ('Name', (2, 17), 'b', ('Store',)))], [('Expr', (2, 20), ('Num', (2, 20), 1))])], [], None)]), ('Module', [('Expr', (1, 0), ('Dict', (1, 0), [None, ('Num', (1, 10), 2)], [('Dict', (1, 3), [('Num', (1, 4), 1)], [('Num', (1, 6), 2)]), ('Num', (1, 12), 3)]))]), ('Module', [('Expr', (1, 0), ('Set', (1, 0), [('Starred', (1, 1), ('Set', (1, 2), [('Num', (1, 3), 1), ('Num', (1, 6), 2)]), ('Load',)), ('Num', (1, 10), 3)]))]), -('Module', [('AsyncFunctionDef', (1, 6), 'f', ('arguments', [], None, [], [], None, []), [('Expr', (2, 1), ('ListComp', (2, 2), ('Name', (2, 2), 'i', ('Load',)), [('comprehension', ('Name', (2, 14), 'b', ('Store',)), ('Name', (2, 19), 'c', ('Load',)), [], 1)]))], [], None)]), +('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Expr', (2, 1), ('ListComp', (2, 2), ('Name', (2, 2), 'i', ('Load',)), [('comprehension', ('Name', (2, 14), 'b', ('Store',)), ('Name', (2, 19), 'c', ('Load',)), [], 1)]))], [], None)]), ] single_results = [ ('Interactive', [('Expr', (1, 0), ('BinOp', (1, 0), ('Num', (1, 0), 1), ('Add',), ('Num', (1, 2), 2)))]), diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-10-30-12-44-50.bpo-31902.a07fa57.rst b/Misc/NEWS.d/next/Core and Builtins/2017-10-30-12-44-50.bpo-31902.a07fa57.rst new file mode 100644 index 000000000000..e2b04b3ae0f5 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2017-10-30-12-44-50.bpo-31902.a07fa57.rst @@ -0,0 +1,3 @@ +Fix the ``col_offset`` attribute for ast nodes ``ast.AsyncFor``, +``ast.AsyncFunctionDef``, and ``ast.AsyncWith``. Previously, ``col_offset`` +pointed to the keyword after ``async``. diff --git a/Python/ast.c b/Python/ast.c index a91c075dae66..ef2c858ae4aa 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -600,8 +600,8 @@ static asdl_seq *ast_for_exprlist(struct compiling *, const node *, static expr_ty ast_for_testlist(struct compiling *, const node *); static stmt_ty ast_for_classdef(struct compiling *, const node *, asdl_seq *); -static stmt_ty ast_for_with_stmt(struct compiling *, const node *, int); -static stmt_ty ast_for_for_stmt(struct compiling *, const node *, int); +static stmt_ty ast_for_with_stmt(struct compiling *, const node *, bool); +static stmt_ty ast_for_for_stmt(struct compiling *, const node *, bool); /* Note different signature for ast_for_call */ static expr_ty ast_for_call(struct compiling *, const node *, expr_ty, bool); @@ -1569,10 +1569,11 @@ ast_for_decorators(struct compiling *c, const node *n) } static stmt_ty -ast_for_funcdef_impl(struct compiling *c, const node *n, - asdl_seq *decorator_seq, int is_async) +ast_for_funcdef_impl(struct compiling *c, const node *n0, + asdl_seq *decorator_seq, bool is_async) { /* funcdef: 'def' NAME parameters ['->' test] ':' suite */ + const node * const n = is_async ? CHILD(n0, 1) : n0; identifier name; arguments_ty args; asdl_seq *body; @@ -1601,7 +1602,7 @@ ast_for_funcdef_impl(struct compiling *c, const node *n, if (is_async) return AsyncFunctionDef(name, args, body, decorator_seq, returns, - LINENO(n), n->n_col_offset, c->c_arena); + LINENO(n), n0->n_col_offset, c->c_arena); else return FunctionDef(name, args, body, decorator_seq, returns, LINENO(n), n->n_col_offset, c->c_arena); @@ -1616,8 +1617,8 @@ ast_for_async_funcdef(struct compiling *c, const node *n, asdl_seq *decorator_se assert(strcmp(STR(CHILD(n, 0)), "async") == 0); REQ(CHILD(n, 1), funcdef); - return ast_for_funcdef_impl(c, CHILD(n, 1), decorator_seq, - 1 /* is_async */); + return ast_for_funcdef_impl(c, n, decorator_seq, + true /* is_async */); } static stmt_ty @@ -1625,7 +1626,7 @@ ast_for_funcdef(struct compiling *c, const node *n, asdl_seq *decorator_seq) { /* funcdef: 'def' NAME parameters ['->' test] ':' suite */ return ast_for_funcdef_impl(c, n, decorator_seq, - 0 /* is_async */); + false /* is_async */); } @@ -1639,15 +1640,15 @@ ast_for_async_stmt(struct compiling *c, const node *n) switch (TYPE(CHILD(n, 1))) { case funcdef: - return ast_for_funcdef_impl(c, CHILD(n, 1), NULL, - 1 /* is_async */); + return ast_for_funcdef_impl(c, n, NULL, + true /* is_async */); case with_stmt: - return ast_for_with_stmt(c, CHILD(n, 1), - 1 /* is_async */); + return ast_for_with_stmt(c, n, + true /* is_async */); case for_stmt: - return ast_for_for_stmt(c, CHILD(n, 1), - 1 /* is_async */); + return ast_for_for_stmt(c, n, + true /* is_async */); default: PyErr_Format(PyExc_SystemError, @@ -3681,8 +3682,9 @@ ast_for_while_stmt(struct compiling *c, const node *n) } static stmt_ty -ast_for_for_stmt(struct compiling *c, const node *n, int is_async) +ast_for_for_stmt(struct compiling *c, const node *n0, bool is_async) { + const node * const n = is_async ? CHILD(n0, 1) : n0; asdl_seq *_target, *seq = NULL, *suite_seq; expr_ty expression; expr_ty target, first; @@ -3717,7 +3719,7 @@ ast_for_for_stmt(struct compiling *c, const node *n, int is_async) if (is_async) return AsyncFor(target, expression, suite_seq, seq, - LINENO(n), n->n_col_offset, + LINENO(n), n0->n_col_offset, c->c_arena); else return For(target, expression, suite_seq, seq, @@ -3869,8 +3871,9 @@ ast_for_with_item(struct compiling *c, const node *n) /* with_stmt: 'with' with_item (',' with_item)* ':' suite */ static stmt_ty -ast_for_with_stmt(struct compiling *c, const node *n, int is_async) +ast_for_with_stmt(struct compiling *c, const node *n0, bool is_async) { + const node * const n = is_async ? CHILD(n0, 1) : n0; int i, n_items; asdl_seq *items, *body; @@ -3892,7 +3895,7 @@ ast_for_with_stmt(struct compiling *c, const node *n, int is_async) return NULL; if (is_async) - return AsyncWith(items, body, LINENO(n), n->n_col_offset, c->c_arena); + return AsyncWith(items, body, LINENO(n), n0->n_col_offset, c->c_arena); else return With(items, body, LINENO(n), n->n_col_offset, c->c_arena); } From webhook-mailer at python.org Tue Sep 11 17:49:17 2018 From: webhook-mailer at python.org (Raymond Hettinger) Date: Tue, 11 Sep 2018 21:49:17 -0000 Subject: [Python-checkins] bpo-20180: convert most of itertoolsmodule.c to use Argument Clinic (GH-9164) Message-ID: https://github.com/python/cpython/commit/c4bccd3c7617018b1ce16f95840ffe1a890d44df commit: c4bccd3c7617018b1ce16f95840ffe1a890d44df branch: master author: Tal Einat committer: Raymond Hettinger date: 2018-09-11T14:49:13-07:00 summary: bpo-20180: convert most of itertoolsmodule.c to use Argument Clinic (GH-9164) files: M Modules/clinic/itertoolsmodule.c.h M Modules/itertoolsmodule.c diff --git a/Modules/clinic/itertoolsmodule.c.h b/Modules/clinic/itertoolsmodule.c.h index 68e67494bc93..94df96c0b7e4 100644 --- a/Modules/clinic/itertoolsmodule.c.h +++ b/Modules/clinic/itertoolsmodule.c.h @@ -61,4 +61,452 @@ itertools__grouper(PyTypeObject *type, PyObject *args, PyObject *kwargs) exit: return return_value; } -/*[clinic end generated code: output=82e10c91569d2b95 input=a9049054013a1b77]*/ + +PyDoc_STRVAR(itertools_teedataobject__doc__, +"teedataobject(iterable, values, next, /)\n" +"--\n" +"\n" +"Data container common to multiple tee objects."); + +static PyObject * +itertools_teedataobject_impl(PyTypeObject *type, PyObject *it, + PyObject *values, PyObject *next); + +static PyObject * +itertools_teedataobject(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *it; + PyObject *values; + PyObject *next; + + if ((type == &teedataobject_type) && + !_PyArg_NoKeywords("teedataobject", kwargs)) { + goto exit; + } + if (!PyArg_ParseTuple(args, "OO!O:teedataobject", + &it, &PyList_Type, &values, &next)) { + goto exit; + } + return_value = itertools_teedataobject_impl(type, it, values, next); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools__tee__doc__, +"_tee(iterable, /)\n" +"--\n" +"\n" +"Iterator wrapped to make it copyable."); + +static PyObject * +itertools__tee_impl(PyTypeObject *type, PyObject *iterable); + +static PyObject * +itertools__tee(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *iterable; + + if ((type == &tee_type) && + !_PyArg_NoKeywords("_tee", kwargs)) { + goto exit; + } + if (!PyArg_UnpackTuple(args, "_tee", + 1, 1, + &iterable)) { + goto exit; + } + return_value = itertools__tee_impl(type, iterable); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_tee__doc__, +"tee($module, iterable, n=2, /)\n" +"--\n" +"\n" +"Returns a tuple of n independent iterators."); + +#define ITERTOOLS_TEE_METHODDEF \ + {"tee", (PyCFunction)itertools_tee, METH_FASTCALL, itertools_tee__doc__}, + +static PyObject * +itertools_tee_impl(PyObject *module, PyObject *iterable, Py_ssize_t n); + +static PyObject * +itertools_tee(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *iterable; + Py_ssize_t n = 2; + + if (!_PyArg_ParseStack(args, nargs, "O|n:tee", + &iterable, &n)) { + goto exit; + } + return_value = itertools_tee_impl(module, iterable, n); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_cycle__doc__, +"cycle(iterable, /)\n" +"--\n" +"\n" +"Return elements from the iterable until it is exhausted. Then repeat the sequence indefinitely."); + +static PyObject * +itertools_cycle_impl(PyTypeObject *type, PyObject *iterable); + +static PyObject * +itertools_cycle(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *iterable; + + if ((type == &cycle_type) && + !_PyArg_NoKeywords("cycle", kwargs)) { + goto exit; + } + if (!PyArg_UnpackTuple(args, "cycle", + 1, 1, + &iterable)) { + goto exit; + } + return_value = itertools_cycle_impl(type, iterable); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_dropwhile__doc__, +"dropwhile(predicate, iterable, /)\n" +"--\n" +"\n" +"Drop items from the iterable while predicate(item) is true.\n" +"\n" +"Afterwards, return every element until the iterable is exhausted."); + +static PyObject * +itertools_dropwhile_impl(PyTypeObject *type, PyObject *func, PyObject *seq); + +static PyObject * +itertools_dropwhile(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *func; + PyObject *seq; + + if ((type == &dropwhile_type) && + !_PyArg_NoKeywords("dropwhile", kwargs)) { + goto exit; + } + if (!PyArg_UnpackTuple(args, "dropwhile", + 2, 2, + &func, &seq)) { + goto exit; + } + return_value = itertools_dropwhile_impl(type, func, seq); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_takewhile__doc__, +"takewhile(predicate, iterable, /)\n" +"--\n" +"\n" +"Return successive entries from an iterable as long as the predicate evaluates to true for each entry."); + +static PyObject * +itertools_takewhile_impl(PyTypeObject *type, PyObject *func, PyObject *seq); + +static PyObject * +itertools_takewhile(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *func; + PyObject *seq; + + if ((type == &takewhile_type) && + !_PyArg_NoKeywords("takewhile", kwargs)) { + goto exit; + } + if (!PyArg_UnpackTuple(args, "takewhile", + 2, 2, + &func, &seq)) { + goto exit; + } + return_value = itertools_takewhile_impl(type, func, seq); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_starmap__doc__, +"starmap(function, iterable, /)\n" +"--\n" +"\n" +"Return an iterator whose values are returned from the function evaluated with an argument tuple taken from the given sequence."); + +static PyObject * +itertools_starmap_impl(PyTypeObject *type, PyObject *func, PyObject *seq); + +static PyObject * +itertools_starmap(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *func; + PyObject *seq; + + if ((type == &starmap_type) && + !_PyArg_NoKeywords("starmap", kwargs)) { + goto exit; + } + if (!PyArg_UnpackTuple(args, "starmap", + 2, 2, + &func, &seq)) { + goto exit; + } + return_value = itertools_starmap_impl(type, func, seq); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_chain_from_iterable__doc__, +"from_iterable($type, iterable, /)\n" +"--\n" +"\n" +"Alternative chain() constructor taking a single iterable argument that evaluates lazily."); + +#define ITERTOOLS_CHAIN_FROM_ITERABLE_METHODDEF \ + {"from_iterable", (PyCFunction)itertools_chain_from_iterable, METH_O|METH_CLASS, itertools_chain_from_iterable__doc__}, + +PyDoc_STRVAR(itertools_combinations__doc__, +"combinations(iterable, r)\n" +"--\n" +"\n" +"Return successive r-length combinations of elements in the iterable.\n" +"\n" +"combinations(range(4), 3) --> (0,1,2), (0,1,3), (0,2,3), (1,2,3)"); + +static PyObject * +itertools_combinations_impl(PyTypeObject *type, PyObject *iterable, + Py_ssize_t r); + +static PyObject * +itertools_combinations(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"iterable", "r", NULL}; + static _PyArg_Parser _parser = {"On:combinations", _keywords, 0}; + PyObject *iterable; + Py_ssize_t r; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &iterable, &r)) { + goto exit; + } + return_value = itertools_combinations_impl(type, iterable, r); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_combinations_with_replacement__doc__, +"combinations_with_replacement(iterable, r)\n" +"--\n" +"\n" +"Return successive r-length combinations of elements in the iterable allowing individual elements to have successive repeats.\n" +"\n" +"combinations_with_replacement(\'ABC\', 2) --> AA AB AC BB BC CC\""); + +static PyObject * +itertools_combinations_with_replacement_impl(PyTypeObject *type, + PyObject *iterable, + Py_ssize_t r); + +static PyObject * +itertools_combinations_with_replacement(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"iterable", "r", NULL}; + static _PyArg_Parser _parser = {"On:combinations_with_replacement", _keywords, 0}; + PyObject *iterable; + Py_ssize_t r; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &iterable, &r)) { + goto exit; + } + return_value = itertools_combinations_with_replacement_impl(type, iterable, r); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_permutations__doc__, +"permutations(iterable, r=None)\n" +"--\n" +"\n" +"Return successive r-length permutations of elements in the iterable.\n" +"\n" +"permutations(range(3), 2) --> (0,1), (0,2), (1,0), (1,2), (2,0), (2,1)"); + +static PyObject * +itertools_permutations_impl(PyTypeObject *type, PyObject *iterable, + PyObject *robj); + +static PyObject * +itertools_permutations(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"iterable", "r", NULL}; + static _PyArg_Parser _parser = {"O|O:permutations", _keywords, 0}; + PyObject *iterable; + PyObject *robj = Py_None; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &iterable, &robj)) { + goto exit; + } + return_value = itertools_permutations_impl(type, iterable, robj); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_accumulate__doc__, +"accumulate(iterable, func=None)\n" +"--\n" +"\n" +"Return series of accumulated sums (or other binary function results)."); + +static PyObject * +itertools_accumulate_impl(PyTypeObject *type, PyObject *iterable, + PyObject *binop); + +static PyObject * +itertools_accumulate(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"iterable", "func", NULL}; + static _PyArg_Parser _parser = {"O|O:accumulate", _keywords, 0}; + PyObject *iterable; + PyObject *binop = Py_None; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &iterable, &binop)) { + goto exit; + } + return_value = itertools_accumulate_impl(type, iterable, binop); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_compress__doc__, +"compress(data, selectors)\n" +"--\n" +"\n" +"Return data elements corresponding to true selector elements.\n" +"\n" +"Forms a shorter iterator from selected data elements using the selectors to\n" +"choose the data elements."); + +static PyObject * +itertools_compress_impl(PyTypeObject *type, PyObject *seq1, PyObject *seq2); + +static PyObject * +itertools_compress(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"data", "selectors", NULL}; + static _PyArg_Parser _parser = {"OO:compress", _keywords, 0}; + PyObject *seq1; + PyObject *seq2; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &seq1, &seq2)) { + goto exit; + } + return_value = itertools_compress_impl(type, seq1, seq2); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_filterfalse__doc__, +"filterfalse(function, iterable, /)\n" +"--\n" +"\n" +"Return those items of iterable for which function(item) is false.\n" +"\n" +"If function is None, return the items that are false."); + +static PyObject * +itertools_filterfalse_impl(PyTypeObject *type, PyObject *func, PyObject *seq); + +static PyObject * +itertools_filterfalse(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *func; + PyObject *seq; + + if ((type == &filterfalse_type) && + !_PyArg_NoKeywords("filterfalse", kwargs)) { + goto exit; + } + if (!PyArg_UnpackTuple(args, "filterfalse", + 2, 2, + &func, &seq)) { + goto exit; + } + return_value = itertools_filterfalse_impl(type, func, seq); + +exit: + return return_value; +} + +PyDoc_STRVAR(itertools_count__doc__, +"count(start=0, step=1)\n" +"--\n" +"\n" +"Return a count object whose .__next__() method returns consecutive values.\n" +"\n" +"Equivalent to:\n" +" def count(firstval=0, step=1):\n" +" x = firstval\n" +" while 1:\n" +" yield x\n" +" x += step"); + +static PyObject * +itertools_count_impl(PyTypeObject *type, PyObject *long_cnt, + PyObject *long_step); + +static PyObject * +itertools_count(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"start", "step", NULL}; + static _PyArg_Parser _parser = {"|OO:count", _keywords, 0}; + PyObject *long_cnt = NULL; + PyObject *long_step = NULL; + + if (!_PyArg_ParseTupleAndKeywordsFast(args, kwargs, &_parser, + &long_cnt, &long_step)) { + goto exit; + } + return_value = itertools_count_impl(type, long_cnt, long_step); + +exit: + return return_value; +} +/*[clinic end generated code: output=d9eb9601bd3296ef input=a9049054013a1b77]*/ diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c index 3ad7e5c80651..ec8f0ae14206 100644 --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -11,11 +11,39 @@ module itertools class itertools.groupby "groupbyobject *" "&groupby_type" class itertools._grouper "_grouperobject *" "&_grouper_type" +class itertools.teedataobject "teedataobject *" "&teedataobject_type" +class itertools._tee "teeobject *" "&tee_type" +class itertools.cycle "cycleobject *" "&cycle_type" +class itertools.dropwhile "dropwhileobject *" "&dropwhile_type" +class itertools.takewhile "takewhileobject *" "&takewhile_type" +class itertools.starmap "starmapobject *" "&starmap_type" +class itertools.chain "chainobject *" "&chain_type" +class itertools.combinations "combinationsobject *" "&combinations_type" +class itertools.combinations_with_replacement "cwr_object *" "&cwr_type" +class itertools.permutations "permutationsobject *" "&permutations_type" +class itertools.accumulate "accumulateobject *" "&accumulate_type" +class itertools.compress "compressobject *" "&compress_type" +class itertools.filterfalse "filterfalseobject *" "&filterfalse_type" +class itertools.count "countobject *" "&count_type" [clinic start generated code]*/ -/*[clinic end generated code: output=da39a3ee5e6b4b0d input=9d506f5bb9177570]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=ea05c93c6d94726a]*/ static PyTypeObject groupby_type; static PyTypeObject _grouper_type; +static PyTypeObject teedataobject_type; +static PyTypeObject tee_type; +static PyTypeObject cycle_type; +static PyTypeObject dropwhile_type; +static PyTypeObject takewhile_type; +static PyTypeObject starmap_type; +static PyTypeObject combinations_type; +static PyTypeObject cwr_type; +static PyTypeObject permutations_type; +static PyTypeObject accumulate_type; +static PyTypeObject compress_type; +static PyTypeObject filterfalse_type; +static PyTypeObject count_type; + #include "clinic/itertoolsmodule.c.h" @@ -539,18 +567,25 @@ teedataobject_reduce(teedataobject *tdo, PyObject *Py_UNUSED(ignored)) tdo->nextlink ? tdo->nextlink : Py_None); } -static PyTypeObject teedataobject_type; +/*[clinic input] + at classmethod +itertools.teedataobject.__new__ + iterable as it: object + values: object(subclass_of='&PyList_Type') + next: object + / +Data container common to multiple tee objects. +[clinic start generated code]*/ static PyObject * -teedataobject_new(PyTypeObject *type, PyObject *args, PyObject *kw) +itertools_teedataobject_impl(PyTypeObject *type, PyObject *it, + PyObject *values, PyObject *next) +/*[clinic end generated code: output=3343ceb07e08df5e input=be60f2fabd2b72ba]*/ { teedataobject *tdo; - PyObject *it, *values, *next; Py_ssize_t i, len; assert(type == &teedataobject_type); - if (!PyArg_ParseTuple(args, "OO!O", &it, &PyList_Type, &values, &next)) - return NULL; tdo = (teedataobject *)teedataobject_newinternal(it); if (!tdo) @@ -592,8 +627,6 @@ static PyMethodDef teedataobject_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(teedataobject_doc, "Data container common to multiple tee objects."); - static PyTypeObject teedataobject_type = { PyVarObject_HEAD_INIT(0, 0) /* Must fill in type value later */ "itertools._tee_dataobject", /* tp_name */ @@ -616,7 +649,7 @@ static PyTypeObject teedataobject_type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - teedataobject_doc, /* tp_doc */ + itertools_teedataobject__doc__, /* tp_doc */ (traverseproc)teedataobject_traverse, /* tp_traverse */ (inquiry)teedataobject_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -633,7 +666,7 @@ static PyTypeObject teedataobject_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - teedataobject_new, /* tp_new */ + itertools_teedataobject, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -716,13 +749,18 @@ tee_fromiterable(PyObject *iterable) return (PyObject *)to; } +/*[clinic input] + at classmethod +itertools._tee.__new__ + iterable: object + / +Iterator wrapped to make it copyable. +[clinic start generated code]*/ + static PyObject * -tee_new(PyTypeObject *type, PyObject *args, PyObject *kw) +itertools__tee_impl(PyTypeObject *type, PyObject *iterable) +/*[clinic end generated code: output=b02d3fd26c810c3f input=adc0779d2afe37a2]*/ { - PyObject *iterable; - - if (!PyArg_UnpackTuple(args, "_tee", 1, 1, &iterable)) - return NULL; return tee_fromiterable(iterable); } @@ -771,9 +809,6 @@ tee_setstate(teeobject *to, PyObject *state) Py_RETURN_NONE; } -PyDoc_STRVAR(teeobject_doc, -"Iterator wrapped to make it copyable"); - static PyMethodDef tee_methods[] = { {"__copy__", (PyCFunction)tee_copy, METH_NOARGS, teecopy_doc}, {"__reduce__", (PyCFunction)tee_reduce, METH_NOARGS, reduce_doc}, @@ -803,7 +838,7 @@ static PyTypeObject tee_type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - teeobject_doc, /* tp_doc */ + itertools__tee__doc__, /* tp_doc */ (traverseproc)tee_traverse, /* tp_traverse */ (inquiry)tee_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -820,19 +855,26 @@ static PyTypeObject tee_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - tee_new, /* tp_new */ + itertools__tee, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; +/*[clinic input] +itertools.tee + iterable: object + n: Py_ssize_t = 2 + / +Returns a tuple of n independent iterators. +[clinic start generated code]*/ + static PyObject * -tee(PyObject *self, PyObject *args) +itertools_tee_impl(PyObject *module, PyObject *iterable, Py_ssize_t n) +/*[clinic end generated code: output=1c64519cd859c2f0 input=c99a1472c425d66d]*/ { - Py_ssize_t i, n=2; - PyObject *it, *iterable, *copyable, *copyfunc, *result; + Py_ssize_t i; + PyObject *it, *copyable, *copyfunc, *result; _Py_IDENTIFIER(__copy__); - if (!PyArg_ParseTuple(args, "O|n", &iterable, &n)) - return NULL; if (n < 0) { PyErr_SetString(PyExc_ValueError, "n must be >= 0"); return NULL; @@ -885,9 +927,6 @@ tee(PyObject *self, PyObject *args) return result; } -PyDoc_STRVAR(tee_doc, -"tee(iterable, n=2) --> tuple of n independent iterators."); - /* cycle object **************************************************************/ @@ -901,20 +940,22 @@ typedef struct { static PyTypeObject cycle_type; +/*[clinic input] + at classmethod +itertools.cycle.__new__ + iterable: object + / +Return elements from the iterable until it is exhausted. Then repeat the sequence indefinitely. +[clinic start generated code]*/ + static PyObject * -cycle_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_cycle_impl(PyTypeObject *type, PyObject *iterable) +/*[clinic end generated code: output=f60e5ec17a45b35c input=9d1d84bcf66e908b]*/ { PyObject *it; - PyObject *iterable; PyObject *saved; cycleobject *lz; - if (type == &cycle_type && !_PyArg_NoKeywords("cycle", kwds)) - return NULL; - - if (!PyArg_UnpackTuple(args, "cycle", 1, 1, &iterable)) - return NULL; - /* Get iterator. */ it = PyObject_GetIter(iterable); if (it == NULL) @@ -1041,12 +1082,6 @@ static PyMethodDef cycle_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(cycle_doc, -"cycle(iterable) --> cycle object\n\ -\n\ -Return elements from the iterable until it is exhausted.\n\ -Then repeat the sequence indefinitely."); - static PyTypeObject cycle_type = { PyVarObject_HEAD_INIT(NULL, 0) "itertools.cycle", /* tp_name */ @@ -1070,7 +1105,7 @@ static PyTypeObject cycle_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - cycle_doc, /* tp_doc */ + itertools_cycle__doc__, /* tp_doc */ (traverseproc)cycle_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -1087,7 +1122,7 @@ static PyTypeObject cycle_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - cycle_new, /* tp_new */ + itertools_cycle, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -1103,19 +1138,24 @@ typedef struct { static PyTypeObject dropwhile_type; +/*[clinic input] + at classmethod +itertools.dropwhile.__new__ + predicate as func: object + iterable as seq: object + / +Drop items from the iterable while predicate(item) is true. + +Afterwards, return every element until the iterable is exhausted. +[clinic start generated code]*/ + static PyObject * -dropwhile_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_dropwhile_impl(PyTypeObject *type, PyObject *func, PyObject *seq) +/*[clinic end generated code: output=92f9d0d89af149e4 input=d39737147c9f0a26]*/ { - PyObject *func, *seq; PyObject *it; dropwhileobject *lz; - if (type == &dropwhile_type && !_PyArg_NoKeywords("dropwhile", kwds)) - return NULL; - - if (!PyArg_UnpackTuple(args, "dropwhile", 2, 2, &func, &seq)) - return NULL; - /* Get iterator. */ it = PyObject_GetIter(seq); if (it == NULL) @@ -1209,12 +1249,6 @@ static PyMethodDef dropwhile_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(dropwhile_doc, -"dropwhile(predicate, iterable) --> dropwhile object\n\ -\n\ -Drop items from the iterable while predicate(item) is true.\n\ -Afterwards, return every element until the iterable is exhausted."); - static PyTypeObject dropwhile_type = { PyVarObject_HEAD_INIT(NULL, 0) "itertools.dropwhile", /* tp_name */ @@ -1238,7 +1272,7 @@ static PyTypeObject dropwhile_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - dropwhile_doc, /* tp_doc */ + itertools_dropwhile__doc__, /* tp_doc */ (traverseproc)dropwhile_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -1255,7 +1289,7 @@ static PyTypeObject dropwhile_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - dropwhile_new, /* tp_new */ + itertools_dropwhile, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -1271,19 +1305,22 @@ typedef struct { static PyTypeObject takewhile_type; +/*[clinic input] + at classmethod +itertools.takewhile.__new__ + predicate as func: object + iterable as seq: object + / +Return successive entries from an iterable as long as the predicate evaluates to true for each entry. +[clinic start generated code]*/ + static PyObject * -takewhile_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_takewhile_impl(PyTypeObject *type, PyObject *func, PyObject *seq) +/*[clinic end generated code: output=bb179ea7864e2ef6 input=ba5255f7519aa119]*/ { - PyObject *func, *seq; PyObject *it; takewhileobject *lz; - if (type == &takewhile_type && !_PyArg_NoKeywords("takewhile", kwds)) - return NULL; - - if (!PyArg_UnpackTuple(args, "takewhile", 2, 2, &func, &seq)) - return NULL; - /* Get iterator. */ it = PyObject_GetIter(seq); if (it == NULL) @@ -1373,11 +1410,6 @@ static PyMethodDef takewhile_reduce_methods[] = { setstate_doc}, {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(takewhile_doc, -"takewhile(predicate, iterable) --> takewhile object\n\ -\n\ -Return successive entries from an iterable as long as the\n\ -predicate evaluates to true for each entry."); static PyTypeObject takewhile_type = { PyVarObject_HEAD_INIT(NULL, 0) @@ -1402,7 +1434,7 @@ static PyTypeObject takewhile_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - takewhile_doc, /* tp_doc */ + itertools_takewhile__doc__, /* tp_doc */ (traverseproc)takewhile_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -1419,7 +1451,7 @@ static PyTypeObject takewhile_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - takewhile_new, /* tp_new */ + itertools_takewhile, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -1693,19 +1725,22 @@ typedef struct { static PyTypeObject starmap_type; +/*[clinic input] + at classmethod +itertools.starmap.__new__ + function as func: object + iterable as seq: object + / +Return an iterator whose values are returned from the function evaluated with an argument tuple taken from the given sequence. +[clinic start generated code]*/ + static PyObject * -starmap_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_starmap_impl(PyTypeObject *type, PyObject *func, PyObject *seq) +/*[clinic end generated code: output=79eeb81d452c6e8d input=844766df6a0d4dad]*/ { - PyObject *func, *seq; PyObject *it; starmapobject *lz; - if (type == &starmap_type && !_PyArg_NoKeywords("starmap", kwds)) - return NULL; - - if (!PyArg_UnpackTuple(args, "starmap", 2, 2, &func, &seq)) - return NULL; - /* Get iterator. */ it = PyObject_GetIter(seq); if (it == NULL) @@ -1776,12 +1811,6 @@ static PyMethodDef starmap_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(starmap_doc, -"starmap(function, sequence) --> starmap object\n\ -\n\ -Return an iterator whose values are returned from the function evaluated\n\ -with an argument tuple taken from the given sequence."); - static PyTypeObject starmap_type = { PyVarObject_HEAD_INIT(NULL, 0) "itertools.starmap", /* tp_name */ @@ -1805,7 +1834,7 @@ static PyTypeObject starmap_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - starmap_doc, /* tp_doc */ + itertools_starmap__doc__, /* tp_doc */ (traverseproc)starmap_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -1822,7 +1851,7 @@ static PyTypeObject starmap_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - starmap_new, /* tp_new */ + itertools_starmap, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -1868,8 +1897,17 @@ chain_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return chain_new_internal(type, source); } +/*[clinic input] + at classmethod +itertools.chain.from_iterable + iterable as arg: object + / +Alternative chain() constructor taking a single iterable argument that evaluates lazily. +[clinic start generated code]*/ + static PyObject * -chain_new_from_iterable(PyTypeObject *type, PyObject *arg) +itertools_chain_from_iterable(PyTypeObject *type, PyObject *arg) +/*[clinic end generated code: output=667ae7a7f7b68654 input=72c39e3a2ca3be85]*/ { PyObject *source; @@ -1985,15 +2023,8 @@ Return a chain object whose .__next__() method returns elements from the\n\ first iterable until it is exhausted, then elements from the next\n\ iterable, until all of the iterables are exhausted."); -PyDoc_STRVAR(chain_from_iterable_doc, -"chain.from_iterable(iterable) --> chain object\n\ -\n\ -Alternate chain() constructor taking a single iterable argument\n\ -that evaluates lazily."); - static PyMethodDef chain_methods[] = { - {"from_iterable", (PyCFunction) chain_new_from_iterable, METH_O | METH_CLASS, - chain_from_iterable_doc}, + ITERTOOLS_CHAIN_FROM_ITERABLE_METHODDEF {"__reduce__", (PyCFunction)chain_reduce, METH_NOARGS, reduce_doc}, {"__setstate__", (PyCFunction)chain_setstate, METH_O, @@ -2417,21 +2448,27 @@ typedef struct { static PyTypeObject combinations_type; + +/*[clinic input] + at classmethod +itertools.combinations.__new__ + iterable: object + r: Py_ssize_t +Return successive r-length combinations of elements in the iterable. + +combinations(range(4), 3) --> (0,1,2), (0,1,3), (0,2,3), (1,2,3) +[clinic start generated code]*/ + static PyObject * -combinations_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_combinations_impl(PyTypeObject *type, PyObject *iterable, + Py_ssize_t r) +/*[clinic end generated code: output=87a689b39c40039c input=06bede09e3da20f8]*/ { combinationsobject *co; Py_ssize_t n; - Py_ssize_t r; PyObject *pool = NULL; - PyObject *iterable = NULL; Py_ssize_t *indices = NULL; Py_ssize_t i; - static char *kwargs[] = {"iterable", "r", NULL}; - - if (!PyArg_ParseTupleAndKeywords(args, kwds, "On:combinations", kwargs, - &iterable, &r)) - return NULL; pool = PySequence_Tuple(iterable); if (pool == NULL) @@ -2666,12 +2703,6 @@ static PyMethodDef combinations_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(combinations_doc, -"combinations(iterable, r) --> combinations object\n\ -\n\ -Return successive r-length combinations of elements in the iterable.\n\n\ -combinations(range(4), 3) --> (0,1,2), (0,1,3), (0,2,3), (1,2,3)"); - static PyTypeObject combinations_type = { PyVarObject_HEAD_INIT(NULL, 0) "itertools.combinations", /* tp_name */ @@ -2695,7 +2726,7 @@ static PyTypeObject combinations_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - combinations_doc, /* tp_doc */ + itertools_combinations__doc__, /* tp_doc */ (traverseproc)combinations_traverse,/* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2712,7 +2743,7 @@ static PyTypeObject combinations_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - combinations_new, /* tp_new */ + itertools_combinations, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -2756,22 +2787,27 @@ typedef struct { static PyTypeObject cwr_type; +/*[clinic input] + at classmethod +itertools.combinations_with_replacement.__new__ + iterable: object + r: Py_ssize_t +Return successive r-length combinations of elements in the iterable allowing individual elements to have successive repeats. + +combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC" +[clinic start generated code]*/ + static PyObject * -cwr_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_combinations_with_replacement_impl(PyTypeObject *type, + PyObject *iterable, + Py_ssize_t r) +/*[clinic end generated code: output=48b26856d4e659ca input=dc2a8c7ba785fad7]*/ { cwrobject *co; Py_ssize_t n; - Py_ssize_t r; PyObject *pool = NULL; - PyObject *iterable = NULL; Py_ssize_t *indices = NULL; Py_ssize_t i; - static char *kwargs[] = {"iterable", "r", NULL}; - - if (!PyArg_ParseTupleAndKeywords(args, kwds, - "On:combinations_with_replacement", - kwargs, &iterable, &r)) - return NULL; pool = PySequence_Tuple(iterable); if (pool == NULL) @@ -2996,13 +3032,6 @@ static PyMethodDef cwr_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(cwr_doc, -"combinations_with_replacement(iterable, r) --> combinations_with_replacement object\n\ -\n\ -Return successive r-length combinations of elements in the iterable\n\ -allowing individual elements to have successive repeats.\n\ -combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC"); - static PyTypeObject cwr_type = { PyVarObject_HEAD_INIT(NULL, 0) "itertools.combinations_with_replacement", /* tp_name */ @@ -3026,7 +3055,7 @@ static PyTypeObject cwr_type = { 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ - cwr_doc, /* tp_doc */ + itertools_combinations_with_replacement__doc__, /* tp_doc */ (traverseproc)cwr_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -3043,7 +3072,7 @@ static PyTypeObject cwr_type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - cwr_new, /* tp_new */ + itertools_combinations_with_replacement, /* tp_new */ PyObject_GC_Del, /* tp_free */ }; @@ -3085,23 +3114,28 @@ typedef struct { static PyTypeObject permutations_type; +/*[clinic input] + at classmethod +itertools.permutations.__new__ + iterable: object + r as robj: object = None +Return successive r-length permutations of elements in the iterable. + +permutations(range(3), 2) --> (0,1), (0,2), (1,0), (1,2), (2,0), (2,1) +[clinic start generated code]*/ + static PyObject * -permutations_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +itertools_permutations_impl(PyTypeObject *type, PyObject *iterable, + PyObject *robj) +/*[clinic end generated code: output=296a72fa76d620ea input=57d0170a4ac0ec7a]*/ { permutationsobject *po; Py_ssize_t n; Py_ssize_t r; - PyObject *robj = Py_None; PyObject *pool = NULL; - PyObject *iterable = NULL; Py_ssize_t *indices = NULL; Py_ssize_t *cycles = NULL; Py_ssize_t i; - static char *kwargs[] = {"iterable", "r", NULL}; - - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O:permutations", kwargs, - &iterable, &robj)) - return NULL; pool = PySequence_Tuple(iterable); if (pool == NULL) @@ -3389,12 +3423,6 @@ static PyMethodDef permuations_methods[] = { {NULL, NULL} /* sentinel */ }; -PyDoc_STRVAR(permutations_doc, -"permutations(iterable[, r]) --> permutations object\n\ -\n\ -Return successive r-length per