Mailman 3 December 2018 - Python-checkins

bpo-35214: Annotate posix calls for clang MSan. (#11389)
by Gregory P. Smith Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/1d300ce1d8238136595c8fea76266a4755… commit: 1d300ce1d8238136595c8fea76266a4755cd73a2 branch: master author: Gregory P. Smith <greg(a)krypto.org> committer: GitHub <noreply(a)github.com> date: 2018-12-30T21:13:02-08:00 summary: bpo-35214: Annotate posix calls for clang MSan. (#11389) It doesn't know the details of a few less common libc functions. files: M Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst M Modules/posixmodule.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst b/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst index 62dee0e37008..fa61f78a9328 100644 --- a/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst +++ b/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst @@ -1,2 +1,2 @@ clang Memory Sanitizer build instrumentation was added to work around false -positives from socket, time, test_io, and test_faulthandler. +positives from posix, socket, time, test_io, and test_faulthandler. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 4ff1694c5833..e5c2a9cfc1ec 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -367,6 +367,10 @@ static int win32_can_symlink = 0; #define HAVE_STRUCT_STAT_ST_FSTYPE 1 #endif +#ifdef _Py_MEMORY_SANITIZER +# include <sanitizer/msan_interface.h> +#endif + #ifdef HAVE_FORK static void run_at_forkers(PyObject *lst, int reverse) @@ -5493,6 +5497,9 @@ os_posix_spawn_impl(PyObject *module, path_t *path, PyObject *argv, PyErr_SetFromErrnoWithFilenameObject(PyExc_OSError, path->object); goto exit; } +#ifdef _Py_MEMORY_SANITIZER + __msan_unpoison(&pid, sizeof(pid)); +#endif result = PyLong_FromPid(pid); exit: @@ -6098,6 +6105,9 @@ os_sched_rr_get_interval_impl(PyObject *module, pid_t pid) posix_error(); return -1.0; } +#ifdef _Py_MEMORY_SANITIZER + __msan_unpoison(&interval, sizeof(interval)); +#endif return (double)interval.tv_sec + 1e-9*interval.tv_nsec; } #endif /* HAVE_SCHED_RR_GET_INTERVAL */ @@ -6568,6 +6578,12 @@ posix_getgrouplist(PyObject *self, PyObject *args) return posix_error(); } +#ifdef _Py_MEMORY_SANITIZER + /* Clang memory sanitizer libc intercepts don't know getgrouplist. */ + __msan_unpoison(&ngroups, sizeof(ngroups)); + __msan_unpoison(groups, ngroups*sizeof(*groups)); +#endif + list = PyList_New(ngroups); if (list == NULL) { PyMem_Del(groups);

1 0

bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (GH-11385)
by Miss Islington (bot) Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/5d2e4b1ff2f01b6aeac2f2f302f363d3ee… commit: 5d2e4b1ff2f01b6aeac2f2f302f363d3eed225fa branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-12-30T20:39:28-08:00 summary: bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (GH-11385) * skip test_constructor under msan. * fix the others as well. * reuse existing related news entry. * typo fix (cherry picked from commit e5796c42c687e1454e84dcc50e6f67db48ff69a0) Co-authored-by: Gregory P. Smith <greg(a)krypto.org> files: M Lib/test/test_io.py M Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 405f2d38b494..6655576b6d4b 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -28,6 +28,7 @@ import random import signal import sys +import sysconfig import threading import time import unittest @@ -59,6 +60,13 @@ def byteslike(*pos, **kw): class EmptyStruct(ctypes.Structure): pass +_cflags = sysconfig.get_config_var('CFLAGS') or '' +_config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' +MEMORY_SANITIZER = ( + '-fsanitize=memory' in _cflags or + '--with-memory-sanitizer' in _config_args +) + def _default_chunk_size(): """Get the default TextIOWrapper chunk size""" with open(__file__, "r", encoding="latin-1") as f: @@ -1496,6 +1504,8 @@ def test_read_on_closed(self): class CBufferedReaderTest(BufferedReaderTest, SizeofTest): tp = io.BufferedReader + @unittest.skipIf(MEMORY_SANITIZER, "MSan defaults to crashing " + "instead of returning NULL for malloc failure.") def test_constructor(self): BufferedReaderTest.test_constructor(self) # The allocation can succeed on 32-bit builds, e.g. with more @@ -1840,6 +1850,8 @@ def test_slow_close_from_thread(self): class CBufferedWriterTest(BufferedWriterTest, SizeofTest): tp = io.BufferedWriter + @unittest.skipIf(MEMORY_SANITIZER, "MSan defaults to crashing " + "instead of returning NULL for malloc failure.") def test_constructor(self): BufferedWriterTest.test_constructor(self) # The allocation can succeed on 32-bit builds, e.g. with more @@ -2314,6 +2326,8 @@ def test_interleaved_readline_write(self): class CBufferedRandomTest(BufferedRandomTest, SizeofTest): tp = io.BufferedRandom + @unittest.skipIf(MEMORY_SANITIZER, "MSan defaults to crashing " + "instead of returning NULL for malloc failure.") def test_constructor(self): BufferedRandomTest.test_constructor(self) # The allocation can succeed on 32-bit builds, e.g. with more diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst b/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst index d2e5457842b7..62dee0e37008 100644 --- a/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst +++ b/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst @@ -1,2 +1,2 @@ clang Memory Sanitizer build instrumentation was added to work around false -positives from socket, time, and test_faulthandler. +positives from socket, time, test_io, and test_faulthandler.

1 0

Fix typos in test_faulthandler skipIfs for ubsan. (GH-11386)
by Miss Islington (bot) Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/89e7979f06d237ef81332b84d8ffd246f1… commit: 89e7979f06d237ef81332b84d8ffd246f15c05e8 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-12-30T20:34:03-08:00 summary: Fix typos in test_faulthandler skipIfs for ubsan. (GH-11386) (cherry picked from commit d6f45b23a59de5782ba38d3c698da9c08aaca11b) Co-authored-by: Gregory P. Smith <greg(a)krypto.org> files: M Lib/test/test_faulthandler.py diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py index 69a6f0ffa016..700b7ad6b885 100644 --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -23,11 +23,11 @@ _cflags = sysconfig.get_config_var('CFLAGS') or '' _config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' UB_SANITIZER = ( - '-fsanitizer=undefined' in _cflags or + '-fsanitize=undefined' in _cflags or '--with-undefined-behavior-sanitizer' in _config_args ) MEMORY_SANITIZER = ( - '-fsanitizer=memory' in _cflags or + '-fsanitize=memory' in _cflags or '--with-memory-sanitizer' in _config_args ) @@ -265,7 +265,7 @@ def test_gil_released(self): 'Segmentation fault') @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, - "sanizer builds change crashing process output.") + "sanitizer builds change crashing process output.") @skip_segfault_on_android def test_enable_file(self): with temporary_filename() as filename: @@ -282,7 +282,7 @@ def test_enable_file(self): @unittest.skipIf(sys.platform == "win32", "subprocess doesn't support pass_fds on Windows") @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, - "sanizer builds change crashing process output.") + "sanitizer builds change crashing process output.") @skip_segfault_on_android def test_enable_fd(self): with tempfile.TemporaryFile('wb+') as fp:

1 0

bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (#11385)
by Gregory P. Smith Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/e5796c42c687e1454e84dcc50e6f67db48… commit: e5796c42c687e1454e84dcc50e6f67db48ff69a0 branch: master author: Gregory P. Smith <greg(a)krypto.org> committer: GitHub <noreply(a)github.com> date: 2018-12-30T20:17:57-08:00 summary: bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (#11385) * skip test_constructor under msan. * fix the others as well. * reuse existing related news entry. * typo fix files: M Lib/test/test_io.py M Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index c3644875103d..d245c5d846ad 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -28,6 +28,7 @@ import random import signal import sys +import sysconfig import threading import time import unittest @@ -59,6 +60,13 @@ def byteslike(*pos, **kw): class EmptyStruct(ctypes.Structure): pass +_cflags = sysconfig.get_config_var('CFLAGS') or '' +_config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' +MEMORY_SANITIZER = ( + '-fsanitize=memory' in _cflags or + '--with-memory-sanitizer' in _config_args +) + def _default_chunk_size(): """Get the default TextIOWrapper chunk size""" with open(__file__, "r", encoding="latin-1") as f: @@ -1496,6 +1504,8 @@ def test_read_on_closed(self): class CBufferedReaderTest(BufferedReaderTest, SizeofTest): tp = io.BufferedReader + @unittest.skipIf(MEMORY_SANITIZER, "MSan defaults to crashing " + "instead of returning NULL for malloc failure.") def test_constructor(self): BufferedReaderTest.test_constructor(self) # The allocation can succeed on 32-bit builds, e.g. with more @@ -1840,6 +1850,8 @@ def test_slow_close_from_thread(self): class CBufferedWriterTest(BufferedWriterTest, SizeofTest): tp = io.BufferedWriter + @unittest.skipIf(MEMORY_SANITIZER, "MSan defaults to crashing " + "instead of returning NULL for malloc failure.") def test_constructor(self): BufferedWriterTest.test_constructor(self) # The allocation can succeed on 32-bit builds, e.g. with more @@ -2314,6 +2326,8 @@ def test_interleaved_readline_write(self): class CBufferedRandomTest(BufferedRandomTest, SizeofTest): tp = io.BufferedRandom + @unittest.skipIf(MEMORY_SANITIZER, "MSan defaults to crashing " + "instead of returning NULL for malloc failure.") def test_constructor(self): BufferedRandomTest.test_constructor(self) # The allocation can succeed on 32-bit builds, e.g. with more diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst b/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst index d2e5457842b7..62dee0e37008 100644 --- a/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst +++ b/Misc/NEWS.d/next/Core and Builtins/2018-12-30-15-36-23.bpo-35214.GWDQcv.rst @@ -1,2 +1,2 @@ clang Memory Sanitizer build instrumentation was added to work around false -positives from socket, time, and test_faulthandler. +positives from socket, time, test_io, and test_faulthandler.

1 0

Fix typos in test_faulthandler skipIfs for ubsan. (GH-11386)
by Gregory P. Smith Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/d6f45b23a59de5782ba38d3c698da9c08a… commit: d6f45b23a59de5782ba38d3c698da9c08aaca11b branch: master author: Gregory P. Smith <greg(a)krypto.org> committer: GitHub <noreply(a)github.com> date: 2018-12-30T20:15:41-08:00 summary: Fix typos in test_faulthandler skipIfs for ubsan. (GH-11386) files: M Lib/test/test_faulthandler.py diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py index 80b8868a0696..6aee22be2e03 100644 --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -23,11 +23,11 @@ _cflags = sysconfig.get_config_var('CFLAGS') or '' _config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' UB_SANITIZER = ( - '-fsanitizer=undefined' in _cflags or + '-fsanitize=undefined' in _cflags or '--with-undefined-behavior-sanitizer' in _config_args ) MEMORY_SANITIZER = ( - '-fsanitizer=memory' in _cflags or + '-fsanitize=memory' in _cflags or '--with-memory-sanitizer' in _config_args ) @@ -265,7 +265,7 @@ def test_gil_released(self): 'Segmentation fault') @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, - "sanizer builds change crashing process output.") + "sanitizer builds change crashing process output.") @skip_segfault_on_android def test_enable_file(self): with temporary_filename() as filename: @@ -282,7 +282,7 @@ def test_enable_file(self): @unittest.skipIf(sys.platform == "win32", "subprocess doesn't support pass_fds on Windows") @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, - "sanizer builds change crashing process output.") + "sanitizer builds change crashing process output.") @skip_segfault_on_android def test_enable_fd(self): with tempfile.TemporaryFile('wb+') as fp:

1 0

bpo-35550: Fix incorrect Solaris define guards (GH-11275)
by Miss Islington (bot) Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/d82344378ad8e471b8ed12fb99807f6835… commit: d82344378ad8e471b8ed12fb99807f68351c5412 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-12-30T18:39:00-08:00 summary: bpo-35550: Fix incorrect Solaris define guards (GH-11275) Python source code uses on several places ifdef sun or defined(sun) without the underscores, which is not standard compliant and shouldn't be used. Defines should check for __sun instead. Reference: http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefi… https://bugs.python.org/issue35550 (cherry picked from commit 6f9bc72c79c3262e5d0f2c0e96b016477399cfb1) Co-authored-by: Jakub Kulík <Kulikjak(a)gmail.com> files: A Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst M Modules/_posixsubprocess.c M Modules/posixmodule.c M Modules/socketmodule.c M Modules/timemodule.c M Python/bootstrap_hash.c diff --git a/Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst b/Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst new file mode 100644 index 000000000000..8a6b90d5970e --- /dev/null +++ b/Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst @@ -0,0 +1 @@ +Fix incorrect Solaris #ifdef checks to look for __sun && __SVR4 instead of sun when compiling. \ No newline at end of file diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 851aa9777168..3cf0683ad98b 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -30,7 +30,7 @@ # define SYS_getdents64 __NR_getdents64 #endif -#if defined(sun) +#if defined(__sun) && defined(__SVR4) /* readdir64 is used to work around Solaris 9 bug 6395699. */ # define readdir readdir64 # define dirent dirent64 diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 383999ea8492..909b06e5a9dd 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -5926,7 +5926,7 @@ os_openpty_impl(PyObject *module) #endif #if defined(HAVE_DEV_PTMX) && !defined(HAVE_OPENPTY) && !defined(HAVE__GETPTY) PyOS_sighandler_t sig_saved; -#ifdef sun +#if defined(__sun) && defined(__SVR4) extern char *ptsname(int fildes); #endif #endif diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 661e1bdf5784..988471e15fbe 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -265,7 +265,7 @@ if_indextoname(index) -- return the corresponding interface name\n\ #endif /* Solaris fails to define this variable at all. */ -#if defined(sun) && !defined(INET_ADDRSTRLEN) +#if (defined(__sun) && defined(__SVR4)) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif diff --git a/Modules/timemodule.c b/Modules/timemodule.c index 5e99bd82d1de..bbfb7db7abde 100644 --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -725,7 +725,7 @@ time_strftime(PyObject *self, PyObject *args) return NULL; } -#if defined(_MSC_VER) || defined(sun) || defined(_AIX) +#if defined(_MSC_VER) || (defined(__sun) && defined(__SVR4)) || defined(_AIX) if (buf.tm_year + 1900 < 1 || 9999 < buf.tm_year + 1900) { PyErr_SetString(PyExc_ValueError, "strftime() requires year in [1; 9999]"); @@ -771,7 +771,7 @@ time_strftime(PyObject *self, PyObject *args) return NULL; } } -#elif (defined(_AIX) || defined(sun)) && defined(HAVE_WCSFTIME) +#elif (defined(_AIX) || (defined(__sun) && defined(__SVR4))) && defined(HAVE_WCSFTIME) for (outbuf = wcschr(fmt, '%'); outbuf != NULL; outbuf = wcschr(outbuf+2, '%')) diff --git a/Python/bootstrap_hash.c b/Python/bootstrap_hash.c index e2afba2b2e33..58b0802f4668 100644 --- a/Python/bootstrap_hash.c +++ b/Python/bootstrap_hash.c @@ -116,7 +116,7 @@ py_getrandom(void *buffer, Py_ssize_t size, int blocking, int raise) flags = blocking ? 0 : GRND_NONBLOCK; dest = buffer; while (0 < size) { -#ifdef sun +#if defined(__sun) && defined(__SVR4) /* Issue #26735: On Solaris, getrandom() is limited to returning up to 1024 bytes. Call it multiple times if more bytes are requested. */ @@ -266,7 +266,7 @@ py_getentropy(char *buffer, Py_ssize_t size, int raise) } return 1; } -#endif /* defined(HAVE_GETENTROPY) && !defined(sun) */ +#endif /* defined(HAVE_GETENTROPY) && !(defined(__sun) && defined(__SVR4)) */ static struct {

1 0

Cleanup test_faulthandler sanitizer skip logic. (GH-11381)
by Miss Islington (bot) Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/e40429905773cdbdb5458a5d93600b5816… commit: e40429905773cdbdb5458a5d93600b5816668470 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-12-30T18:28:11-08:00 summary: Cleanup test_faulthandler sanitizer skip logic. (GH-11381) Also skip the same tests when using the undefined behavior sanitizer as they much with the output. Updates a regex in another test to use multi-line mode so that the ubsan buildbot should pass again rather than also adding a skip to that one. (cherry picked from commit 30e023256aa1b00d4c783553752fc6f2cc0b9b27) Co-authored-by: Gregory P. Smith <greg(a)krypto.org> files: M Lib/test/test_faulthandler.py diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py index 984690f248ba..69a6f0ffa016 100644 --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -20,11 +20,18 @@ TIMEOUT = 0.5 MS_WINDOWS = (os.name == 'nt') +_cflags = sysconfig.get_config_var('CFLAGS') or '' +_config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' +UB_SANITIZER = ( + '-fsanitizer=undefined' in _cflags or + '--with-undefined-behavior-sanitizer' in _config_args +) MEMORY_SANITIZER = ( - sysconfig.get_config_var("CONFIG_ARGS") and - ("--with-memory-sanitizer" in sysconfig.get_config_var("CONFIG_ARGS")) + '-fsanitizer=memory' in _cflags or + '--with-memory-sanitizer' in _config_args ) + def expected_traceback(lineno1, lineno2, header, min_count=1): regex = header regex += ' File "<string>", line %s in func\n' % lineno1 @@ -99,7 +106,7 @@ def check_error(self, code, line_number, fatal_error, *, else: header = 'Stack' regex = r""" - ^{fatal_error} + (?m)^{fatal_error} {header} $most recent call first$: File "<string>", line {lineno} in <module> @@ -257,8 +264,8 @@ def test_gil_released(self): 3, 'Segmentation fault') - @unittest.skipIf(MEMORY_SANITIZER, - "memory-sanizer builds change crashing process output.") + @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, + "sanizer builds change crashing process output.") @skip_segfault_on_android def test_enable_file(self): with temporary_filename() as filename: @@ -274,8 +281,8 @@ def test_enable_file(self): @unittest.skipIf(sys.platform == "win32", "subprocess doesn't support pass_fds on Windows") - @unittest.skipIf(MEMORY_SANITIZER, - "memory-sanizer builds change crashing process output.") + @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, + "sanizer builds change crashing process output.") @skip_segfault_on_android def test_enable_fd(self): with tempfile.TemporaryFile('wb+') as fp:

1 0

bpo-35550: Fix incorrect Solaris define guards (GH-11275)
by Gregory P. Smith Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/6f9bc72c79c3262e5d0f2c0e96b0164773… commit: 6f9bc72c79c3262e5d0f2c0e96b016477399cfb1 branch: master author: Jakub Kulík <Kulikjak(a)gmail.com> committer: Gregory P. Smith <greg(a)krypto.org> date: 2018-12-30T18:16:40-08:00 summary: bpo-35550: Fix incorrect Solaris define guards (GH-11275) Python source code uses on several places ifdef sun or defined(sun) without the underscores, which is not standard compliant and shouldn't be used. Defines should check for __sun instead. Reference: http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefi… https://bugs.python.org/issue35550 files: A Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst M Modules/_posixsubprocess.c M Modules/posixmodule.c M Modules/socketmodule.c M Modules/timemodule.c M Python/bootstrap_hash.c diff --git a/Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst b/Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst new file mode 100644 index 000000000000..8a6b90d5970e --- /dev/null +++ b/Misc/NEWS.d/next/Build/2018-12-29-10-19-43.bpo-35550.BTuu8e.rst @@ -0,0 +1 @@ +Fix incorrect Solaris #ifdef checks to look for __sun && __SVR4 instead of sun when compiling. \ No newline at end of file diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 2b2c57d04185..81a23c6d3300 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -30,7 +30,7 @@ # define SYS_getdents64 __NR_getdents64 #endif -#if defined(sun) +#if defined(__sun) && defined(__SVR4) /* readdir64 is used to work around Solaris 9 bug 6395699. */ # define readdir readdir64 # define dirent dirent64 diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 41fedb097ed0..4ff1694c5833 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -6335,7 +6335,7 @@ os_openpty_impl(PyObject *module) #endif #if defined(HAVE_DEV_PTMX) && !defined(HAVE_OPENPTY) && !defined(HAVE__GETPTY) PyOS_sighandler_t sig_saved; -#ifdef sun +#if defined(__sun) && defined(__SVR4) extern char *ptsname(int fildes); #endif #endif diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 0ae280f29881..8c3c2faddb9d 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -267,7 +267,7 @@ if_indextoname(index) -- return the corresponding interface name\n\ #endif /* Solaris fails to define this variable at all. */ -#if defined(sun) && !defined(INET_ADDRSTRLEN) +#if (defined(__sun) && defined(__SVR4)) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif diff --git a/Modules/timemodule.c b/Modules/timemodule.c index 43951d5623d3..fa0f1982c65e 100644 --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -755,7 +755,7 @@ time_strftime(PyObject *self, PyObject *args) return NULL; } -#if defined(_MSC_VER) || defined(sun) || defined(_AIX) +#if defined(_MSC_VER) || (defined(__sun) && defined(__SVR4)) || defined(_AIX) if (buf.tm_year + 1900 < 1 || 9999 < buf.tm_year + 1900) { PyErr_SetString(PyExc_ValueError, "strftime() requires year in [1; 9999]"); @@ -801,7 +801,7 @@ time_strftime(PyObject *self, PyObject *args) return NULL; } } -#elif (defined(_AIX) || defined(sun)) && defined(HAVE_WCSFTIME) +#elif (defined(_AIX) || (defined(__sun) && defined(__SVR4))) && defined(HAVE_WCSFTIME) for (outbuf = wcschr(fmt, '%'); outbuf != NULL; outbuf = wcschr(outbuf+2, '%')) diff --git a/Python/bootstrap_hash.c b/Python/bootstrap_hash.c index eb848c8ff6e3..35d9b7f24a91 100644 --- a/Python/bootstrap_hash.c +++ b/Python/bootstrap_hash.c @@ -114,7 +114,7 @@ py_getrandom(void *buffer, Py_ssize_t size, int blocking, int raise) flags = blocking ? 0 : GRND_NONBLOCK; dest = buffer; while (0 < size) { -#ifdef sun +#if defined(__sun) && defined(__SVR4) /* Issue #26735: On Solaris, getrandom() is limited to returning up to 1024 bytes. Call it multiple times if more bytes are requested. */ @@ -264,7 +264,7 @@ py_getentropy(char *buffer, Py_ssize_t size, int raise) } return 1; } -#endif /* defined(HAVE_GETENTROPY) && !defined(sun) */ +#endif /* defined(HAVE_GETENTROPY) && !(defined(__sun) && defined(__SVR4)) */ static struct {

1 0

Cleanup test_faulthandler sanitizer skip logic. (GH-11381)
by Gregory P. Smith Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/30e023256aa1b00d4c783553752fc6f2cc… commit: 30e023256aa1b00d4c783553752fc6f2cc0b9b27 branch: master author: Gregory P. Smith <greg(a)krypto.org> committer: GitHub <noreply(a)github.com> date: 2018-12-30T18:09:26-08:00 summary: Cleanup test_faulthandler sanitizer skip logic. (GH-11381) Also skip the same tests when using the undefined behavior sanitizer as they much with the output. Updates a regex in another test to use multi-line mode so that the ubsan buildbot should pass again rather than also adding a skip to that one. files: M Lib/test/test_faulthandler.py diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py index 76dd6ab06805..80b8868a0696 100644 --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -20,11 +20,18 @@ TIMEOUT = 0.5 MS_WINDOWS = (os.name == 'nt') +_cflags = sysconfig.get_config_var('CFLAGS') or '' +_config_args = sysconfig.get_config_var('CONFIG_ARGS') or '' +UB_SANITIZER = ( + '-fsanitizer=undefined' in _cflags or + '--with-undefined-behavior-sanitizer' in _config_args +) MEMORY_SANITIZER = ( - sysconfig.get_config_var("CONFIG_ARGS") and - ("--with-memory-sanitizer" in sysconfig.get_config_var("CONFIG_ARGS")) + '-fsanitizer=memory' in _cflags or + '--with-memory-sanitizer' in _config_args ) + def expected_traceback(lineno1, lineno2, header, min_count=1): regex = header regex += ' File "<string>", line %s in func\n' % lineno1 @@ -99,7 +106,7 @@ def check_error(self, code, line_number, fatal_error, *, else: header = 'Stack' regex = r""" - ^{fatal_error} + (?m)^{fatal_error} {header} $most recent call first$: File "<string>", line {lineno} in <module> @@ -257,8 +264,8 @@ def test_gil_released(self): 3, 'Segmentation fault') - @unittest.skipIf(MEMORY_SANITIZER, - "memory-sanizer builds change crashing process output.") + @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, + "sanizer builds change crashing process output.") @skip_segfault_on_android def test_enable_file(self): with temporary_filename() as filename: @@ -274,8 +281,8 @@ def test_enable_file(self): @unittest.skipIf(sys.platform == "win32", "subprocess doesn't support pass_fds on Windows") - @unittest.skipIf(MEMORY_SANITIZER, - "memory-sanizer builds change crashing process output.") + @unittest.skipIf(UB_SANITIZER or MEMORY_SANITIZER, + "sanizer builds change crashing process output.") @skip_segfault_on_android def test_enable_fd(self): with tempfile.TemporaryFile('wb+') as fp:

1 0

bpo-28503: Use crypt_r() when available instead of crypt() (GH-11373) (GH-11376)
by Gregory P. Smith Dec. 31, 2018

Dec. 31, 2018

https://github.com/python/cpython/commit/a144feeb7ec501aaf30072d50e70d54b20… commit: a144feeb7ec501aaf30072d50e70d54b200e5ef0 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Gregory P. Smith <greg(a)krypto.org> date: 2018-12-30T17:59:52-08:00 summary: bpo-28503: Use crypt_r() when available instead of crypt() (GH-11373) (GH-11376) Use crypt_r() when available instead of crypt() in the crypt module. As a nice side effect: This also avoids a memory sanitizer flake as clang msan doesn't know about crypt's internal libc allocated buffer. (cherry picked from commit 387512c7ecde6446f2e29408af2e16b9fc043807) Co-authored-by: Gregory P. Smith <greg(a)krypto.org> [Google] files: A Misc/NEWS.d/next/Library/2018-12-30-14-56-33.bpo-28503.V4kNN3.rst M Include/Python.h M Modules/_cryptmodule.c M configure M configure.ac M pyconfig.h.in diff --git a/Include/Python.h b/Include/Python.h index 05113735c156..54ea3214810e 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -36,7 +36,17 @@ #include <unistd.h> #endif #ifdef HAVE_CRYPT_H +#if defined(HAVE_CRYPT_R) && !defined(_GNU_SOURCE) +/* Required for glibc to expose the crypt_r() function prototype. */ +# define _GNU_SOURCE +# define _Py_GNU_SOURCE_FOR_CRYPT +#endif #include <crypt.h> +#ifdef _Py_GNU_SOURCE_FOR_CRYPT +/* Don't leak the _GNU_SOURCE define to other headers. */ +# undef _GNU_SOURCE +# undef _Py_GNU_SOURCE_FOR_CRYPT +#endif #endif /* For size_t? */ diff --git a/Misc/NEWS.d/next/Library/2018-12-30-14-56-33.bpo-28503.V4kNN3.rst b/Misc/NEWS.d/next/Library/2018-12-30-14-56-33.bpo-28503.V4kNN3.rst new file mode 100644 index 000000000000..651fef1f0371 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-12-30-14-56-33.bpo-28503.V4kNN3.rst @@ -0,0 +1,2 @@ +The `crypt` module now internally uses the `crypt_r()` library function +instead of `crypt()` when available. diff --git a/Modules/_cryptmodule.c b/Modules/_cryptmodule.c index 58d179e6a3d2..5d03f45f6436 100644 --- a/Modules/_cryptmodule.c +++ b/Modules/_cryptmodule.c @@ -34,7 +34,15 @@ static PyObject * crypt_crypt_impl(PyObject *module, const char *word, const char *salt) /*[clinic end generated code: output=0512284a03d2803c input=0e8edec9c364352b]*/ { - return Py_BuildValue("s", crypt(word, salt)); + char *crypt_result; +#ifdef HAVE_CRYPT_R + struct crypt_data data; + memset(&data, 0, sizeof(data)); + crypt_result = crypt_r(word, salt, &data); +#else + crypt_result = crypt(word, salt); +#endif + return Py_BuildValue("s", crypt_result); } diff --git a/configure b/configure index 5e5f974b0e50..7afd48966f61 100755 --- a/configure +++ b/configure @@ -12727,6 +12727,150 @@ fi done +# We search for both crypt and crypt_r as one or the other may be defined +# This gets us our -lcrypt in LIBS when required on the target platform. +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing crypt" >&5 +$as_echo_n "checking for library containing crypt... " >&6; } +if ${ac_cv_search_crypt+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_func_search_save_LIBS=$LIBS +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char crypt (); +int +main () +{ +return crypt (); + ; + return 0; +} +_ACEOF +for ac_lib in '' crypt; do + if test -z "$ac_lib"; then + ac_res="none required" + else + ac_res=-l$ac_lib + LIBS="-l$ac_lib $ac_func_search_save_LIBS" + fi + if ac_fn_c_try_link "$LINENO"; then : + ac_cv_search_crypt=$ac_res +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext + if ${ac_cv_search_crypt+:} false; then : + break +fi +done +if ${ac_cv_search_crypt+:} false; then : + +else + ac_cv_search_crypt=no +fi +rm conftest.$ac_ext +LIBS=$ac_func_search_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_search_crypt" >&5 +$as_echo "$ac_cv_search_crypt" >&6; } +ac_res=$ac_cv_search_crypt +if test "$ac_res" != no; then : + test "$ac_res" = "none required" || LIBS="$ac_res $LIBS" + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing crypt_r" >&5 +$as_echo_n "checking for library containing crypt_r... " >&6; } +if ${ac_cv_search_crypt_r+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_func_search_save_LIBS=$LIBS +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char crypt_r (); +int +main () +{ +return crypt_r (); + ; + return 0; +} +_ACEOF +for ac_lib in '' crypt; do + if test -z "$ac_lib"; then + ac_res="none required" + else + ac_res=-l$ac_lib + LIBS="-l$ac_lib $ac_func_search_save_LIBS" + fi + if ac_fn_c_try_link "$LINENO"; then : + ac_cv_search_crypt_r=$ac_res +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext + if ${ac_cv_search_crypt_r+:} false; then : + break +fi +done +if ${ac_cv_search_crypt_r+:} false; then : + +else + ac_cv_search_crypt_r=no +fi +rm conftest.$ac_ext +LIBS=$ac_func_search_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_search_crypt_r" >&5 +$as_echo "$ac_cv_search_crypt_r" >&6; } +ac_res=$ac_cv_search_crypt_r +if test "$ac_res" != no; then : + test "$ac_res" = "none required" || LIBS="$ac_res $LIBS" + +fi + + +ac_fn_c_check_func "$LINENO" "crypt_r" "ac_cv_func_crypt_r" +if test "x$ac_cv_func_crypt_r" = xyes; then : + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +#define _GNU_SOURCE /* Required for crypt_r()'s prototype in glibc. */ +#include <crypt.h> + +int +main () +{ + +struct crypt_data d; +char *r = crypt_r("", "", &d); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_compile "$LINENO"; then : + +$as_echo "#define HAVE_CRYPT_R 1" >>confdefs.h + +fi +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext + +fi + + for ac_func in clock_gettime do : ac_fn_c_check_func "$LINENO" "clock_gettime" "ac_cv_func_clock_gettime" diff --git a/configure.ac b/configure.ac index a7de901e08b7..788c361ffeeb 100644 --- a/configure.ac +++ b/configure.ac @@ -3875,6 +3875,23 @@ AC_CHECK_FUNCS(gettimeofday, ]) ) +# We search for both crypt and crypt_r as one or the other may be defined +# This gets us our -lcrypt in LIBS when required on the target platform. +AC_SEARCH_LIBS(crypt, crypt) +AC_SEARCH_LIBS(crypt_r, crypt) + +AC_CHECK_FUNC(crypt_r, + AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[ +#define _GNU_SOURCE /* Required for crypt_r()'s prototype in glibc. */ +#include <crypt.h> +]], [[ +struct crypt_data d; +char *r = crypt_r("", "", &d); +]])], + [AC_DEFINE(HAVE_CRYPT_R, 1, [Define if you have the crypt_r() function.])], + []) +) + AC_CHECK_FUNCS(clock_gettime, [], [ AC_CHECK_LIB(rt, clock_gettime, [ LIBS="$LIBS -lrt" diff --git a/pyconfig.h.in b/pyconfig.h.in index a82e82cc9ccc..4032fa519f8a 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -143,6 +143,9 @@ /* Define to 1 if you have the <crypt.h> header file. */ #undef HAVE_CRYPT_H +/* Define if you have the crypt_r() function. */ +#undef HAVE_CRYPT_R + /* Define to 1 if you have the `ctermid' function. */ #undef HAVE_CTERMID

1 0