Mailman 3 November 2019 - Python-checkins

closes bpo-29275: Remove Y2K reference from time module docs (GH-17321)
by Benjamin Peterson 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/42bc60ead39c7be9f6bb7329977826e962… commit: 42bc60ead39c7be9f6bb7329977826e962f601eb branch: master author: Callum Ward <wards.callum(a)gmail.com> committer: Benjamin Peterson <benjamin(a)python.org> date: 2019-11-22T08:57:14-08:00 summary: closes bpo-29275: Remove Y2K reference from time module docs (GH-17321) The Y2K reference is not needed as it only points out that Python's use of C standard functions doesn't generally suffer from Y2K issues; the point regarding conventions for conversion of 2-digit years in :func:`strptime` is still valid. files: M Doc/library/time.rst diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 65ab679669eb3..e628ac44e8064 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -42,17 +42,12 @@ An explanation of some terminology and conventions is in order. library; for 32-bit systems, it is typically in 2038. .. index:: - single: Year 2000 - single: Y2K + single: 2-digit years -.. _time-y2kissues: - -* **Year 2000 (Y2K) issues**: Python depends on the platform's C library, which - generally doesn't have year 2000 issues, since all dates and times are - represented internally as seconds since the epoch. Function :func:`strptime` - can parse 2-digit years when given ``%y`` format code. When 2-digit years are - parsed, they are converted according to the POSIX and ISO C standards: values - 69--99 are mapped to 1969--1999, and values 0--68 are mapped to 2000--2068. +* Function :func:`strptime` can parse 2-digit years when given ``%y`` format + code. When 2-digit years are parsed, they are converted according to the POSIX + and ISO C standards: values 69--99 are mapped to 1969--1999, and values 0--68 + are mapped to 2000--2068. .. index:: single: UTC

1 0

bpo-38858: Add init_interp_main() subfunction (GH-17347)
by Victor Stinner 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/b00513636c9891deba4cae50217e25e8fa… commit: b00513636c9891deba4cae50217e25e8faf6c6ac branch: master author: Victor Stinner <vstinner(a)python.org> committer: GitHub <noreply(a)github.com> date: 2019-11-22T17:52:42+01:00 summary: bpo-38858: Add init_interp_main() subfunction (GH-17347) Fix new_interpreter() error handling: undo it all if status is an exception. files: M Python/pylifecycle.c diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 4825b8b28e621..e692d75999d6b 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -260,6 +260,7 @@ _Py_LegacyLocaleDetected(int warn) #endif } +#ifndef MS_WINDOWS static const char *_C_LOCALE_WARNING = "Python runtime initialized with LC_CTYPE=C (a locale with default ASCII " "encoding), which may cause Unicode compatibility problems. Using C.UTF-8, " @@ -274,6 +275,7 @@ emit_stderr_warning_for_legacy_locale(_PyRuntimeState *runtime) PySys_FormatStderr("%s", _C_LOCALE_WARNING); } } +#endif /* !defined(MS_WINDOWS) */ typedef struct _CandidateLocale { const char *locale_name; /* The locale to try as a coercion target */ @@ -896,16 +898,16 @@ pyinit_core(_PyRuntimeState *runtime, configuration. Example of bpo-34008: Py_Main() called after Py_Initialize(). */ static PyStatus -_Py_ReconfigureMainInterpreter(PyInterpreterState *interp) +_Py_ReconfigureMainInterpreter(PyThreadState *tstate) { - PyConfig *config = &interp->config; + PyConfig *config = &tstate->interp->config; PyObject *argv = _PyWideStringList_AsList(&config->argv); if (argv == NULL) { return _PyStatus_NO_MEMORY(); \ } - int res = PyDict_SetItemString(interp->sysdict, "argv", argv); + int res = PyDict_SetItemString(tstate->interp->sysdict, "argv", argv); Py_DECREF(argv); if (res < 0) { return _PyStatus_ERR("fail to set sys.argv"); @@ -913,60 +915,48 @@ _Py_ReconfigureMainInterpreter(PyInterpreterState *interp) return _PyStatus_OK(); } -/* Update interpreter state based on supplied configuration settings - * - * After calling this function, most of the restrictions on the interpreter - * are lifted. The only remaining incomplete settings are those related - * to the main module (sys.argv[0], __main__ metadata) - * - * Calling this when the interpreter is not initializing, is already - * initialized or without a valid current thread state is a fatal error. - * Other errors should be reported as normal Python exceptions with a - * non-zero return code. - */ + static PyStatus -pyinit_main(PyThreadState *tstate) +init_interp_main(PyThreadState *tstate) { - _PyRuntimeState *runtime = tstate->interp->runtime; - if (!runtime->core_initialized) { - return _PyStatus_ERR("runtime core not initialized"); - } - - /* Configure the main interpreter */ + PyStatus status; + int is_main_interp = _Py_IsMainInterpreter(tstate); PyInterpreterState *interp = tstate->interp; PyConfig *config = &interp->config; - if (runtime->initialized) { - return _Py_ReconfigureMainInterpreter(interp); - } - if (!config->_install_importlib) { /* Special mode for freeze_importlib: run with no import system * * This means anything which needs support from extension modules * or pure Python code in the standard library won't work. */ - runtime->initialized = 1; + if (is_main_interp) { + interp->runtime->initialized = 1; + } return _PyStatus_OK(); } - if (_PyTime_Init() < 0) { - return _PyStatus_ERR("can't initialize time"); - } + if (is_main_interp) { + if (_PyTime_Init() < 0) { + return _PyStatus_ERR("can't initialize time"); + } - if (_PySys_InitMain(tstate) < 0) { - return _PyStatus_ERR("can't finish initializing sys"); + if (_PySys_InitMain(tstate) < 0) { + return _PyStatus_ERR("can't finish initializing sys"); + } } - PyStatus status = init_importlib_external(tstate); + status = init_importlib_external(tstate); if (_PyStatus_EXCEPTION(status)) { return status; } - /* initialize the faulthandler module */ - status = _PyFaulthandler_Init(config->faulthandler); - if (_PyStatus_EXCEPTION(status)) { - return status; + if (is_main_interp) { + /* initialize the faulthandler module */ + status = _PyFaulthandler_Init(config->faulthandler); + if (_PyStatus_EXCEPTION(status)) { + return status; + } } status = _PyUnicode_InitEncodings(tstate); @@ -974,45 +964,49 @@ pyinit_main(PyThreadState *tstate) return status; } - if (config->install_signal_handlers) { - status = init_signals(tstate); - if (_PyStatus_EXCEPTION(status)) { - return status; + if (is_main_interp) { + if (config->install_signal_handlers) { + status = init_signals(tstate); + if (_PyStatus_EXCEPTION(status)) { + return status; + } } - } - if (_PyTraceMalloc_Init(config->tracemalloc) < 0) { - return _PyStatus_ERR("can't initialize tracemalloc"); + if (_PyTraceMalloc_Init(config->tracemalloc) < 0) { + return _PyStatus_ERR("can't initialize tracemalloc"); + } } - status = add_main_module(interp); + status = init_sys_streams(tstate); if (_PyStatus_EXCEPTION(status)) { return status; } - status = init_sys_streams(tstate); + status = init_set_builtins_open(tstate); if (_PyStatus_EXCEPTION(status)) { return status; } - status = init_set_builtins_open(tstate); + status = add_main_module(interp); if (_PyStatus_EXCEPTION(status)) { return status; } - /* Initialize warnings. */ - PyObject *warnoptions = PySys_GetObject("warnoptions"); - if (warnoptions != NULL && PyList_Size(warnoptions) > 0) - { - PyObject *warnings_module = PyImport_ImportModule("warnings"); - if (warnings_module == NULL) { - fprintf(stderr, "'import warnings' failed; traceback:\n"); - _PyErr_Print(tstate); + if (is_main_interp) { + /* Initialize warnings. */ + PyObject *warnoptions = PySys_GetObject("warnoptions"); + if (warnoptions != NULL && PyList_Size(warnoptions) > 0) + { + PyObject *warnings_module = PyImport_ImportModule("warnings"); + if (warnings_module == NULL) { + fprintf(stderr, "'import warnings' failed; traceback:\n"); + _PyErr_Print(tstate); + } + Py_XDECREF(warnings_module); } - Py_XDECREF(warnings_module); - } - runtime->initialized = 1; + interp->runtime->initialized = 1; + } if (config->site_import) { status = init_import_site(); @@ -1021,14 +1015,47 @@ pyinit_main(PyThreadState *tstate) } } + if (is_main_interp) { #ifndef MS_WINDOWS - emit_stderr_warning_for_legacy_locale(runtime); + emit_stderr_warning_for_legacy_locale(interp->runtime); #endif + } return _PyStatus_OK(); } +/* Update interpreter state based on supplied configuration settings + * + * After calling this function, most of the restrictions on the interpreter + * are lifted. The only remaining incomplete settings are those related + * to the main module (sys.argv[0], __main__ metadata) + * + * Calling this when the interpreter is not initializing, is already + * initialized or without a valid current thread state is a fatal error. + * Other errors should be reported as normal Python exceptions with a + * non-zero return code. + */ +static PyStatus +pyinit_main(PyThreadState *tstate) +{ + PyInterpreterState *interp = tstate->interp; + if (!interp->runtime->core_initialized) { + return _PyStatus_ERR("runtime core not initialized"); + } + + if (interp->runtime->initialized) { + return _Py_ReconfigureMainInterpreter(tstate); + } + + PyStatus status = init_interp_main(tstate); + if (_PyStatus_EXCEPTION(status)) { + return status; + } + return _PyStatus_OK(); +} + + PyStatus _Py_InitializeMain(void) { @@ -1440,6 +1467,7 @@ Py_Finalize(void) Py_FinalizeEx(); } + /* Create and initialize a new interpreter and thread, and return the new thread. This requires that Py_Initialize() has been called first. @@ -1499,7 +1527,7 @@ new_interpreter(PyThreadState **tstate_p) status = _PyConfig_Copy(&interp->config, config); if (_PyStatus_EXCEPTION(status)) { - return status; + goto done; } config = &interp->config; @@ -1508,7 +1536,8 @@ new_interpreter(PyThreadState **tstate_p) /* XXX The following is lax in error checking */ PyObject *modules = PyDict_New(); if (modules == NULL) { - return _PyStatus_ERR("can't make modules dictionary"); + status = _PyStatus_ERR("can't make modules dictionary"); + goto done; } interp->modules = modules; @@ -1516,101 +1545,78 @@ new_interpreter(PyThreadState **tstate_p) if (sysmod != NULL) { interp->sysdict = PyModule_GetDict(sysmod); if (interp->sysdict == NULL) { - goto handle_error; + goto handle_exc; } Py_INCREF(interp->sysdict); PyDict_SetItemString(interp->sysdict, "modules", modules); if (_PySys_InitMain(tstate) < 0) { - return _PyStatus_ERR("can't finish initializing sys"); + status = _PyStatus_ERR("can't finish initializing sys"); + goto done; } } else if (_PyErr_Occurred(tstate)) { - goto handle_error; + goto handle_exc; } PyObject *bimod = _PyImport_FindBuiltin(tstate, "builtins"); if (bimod != NULL) { interp->builtins = PyModule_GetDict(bimod); if (interp->builtins == NULL) - goto handle_error; + goto handle_exc; Py_INCREF(interp->builtins); } else if (_PyErr_Occurred(tstate)) { - goto handle_error; + goto handle_exc; } if (bimod != NULL && sysmod != NULL) { status = _PyBuiltins_AddExceptions(bimod); if (_PyStatus_EXCEPTION(status)) { - return status; + goto done; } status = _PySys_SetPreliminaryStderr(interp->sysdict); if (_PyStatus_EXCEPTION(status)) { - return status; + goto done; } status = _PyImportHooks_Init(tstate); if (_PyStatus_EXCEPTION(status)) { - return status; + goto done; } status = init_importlib(tstate, sysmod); if (_PyStatus_EXCEPTION(status)) { - return status; + goto done; } - status = init_importlib_external(tstate); + status = init_interp_main(tstate); if (_PyStatus_EXCEPTION(status)) { - return status; - } - - status = _PyUnicode_InitEncodings(tstate); - if (_PyStatus_EXCEPTION(status)) { - return status; - } - - status = init_sys_streams(tstate); - if (_PyStatus_EXCEPTION(status)) { - return status; - } - - status = init_set_builtins_open(tstate); - if (_PyStatus_EXCEPTION(status)) { - return status; - } - - status = add_main_module(interp); - if (_PyStatus_EXCEPTION(status)) { - return status; - } - - if (config->site_import) { - status = init_import_site(); - if (_PyStatus_EXCEPTION(status)) { - return status; - } + goto done; } } if (_PyErr_Occurred(tstate)) { - goto handle_error; + goto handle_exc; } *tstate_p = tstate; return _PyStatus_OK(); -handle_error: - /* Oops, it didn't work. Undo it all. */ +handle_exc: + status = _PyStatus_OK(); +done: + *tstate_p = NULL; + + /* Oops, it didn't work. Undo it all. */ PyErr_PrintEx(0); PyThreadState_Clear(tstate); PyThreadState_Delete(tstate); PyInterpreterState_Delete(interp); PyThreadState_Swap(save_tstate); - *tstate_p = NULL; - return _PyStatus_OK(); + return status; } PyThreadState *

1 0

bpo-38858: Add init_set_builtins_open() subfunction (GH-17346)
by Victor Stinner 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/e0c9ab8e26d1648b870b80c296b2490a5e… commit: e0c9ab8e26d1648b870b80c296b2490a5e9553e5 branch: master author: Victor Stinner <vstinner(a)python.org> committer: GitHub <noreply(a)github.com> date: 2019-11-22T16:19:14+01:00 summary: bpo-38858: Add init_set_builtins_open() subfunction (GH-17346) files: M Python/pylifecycle.c diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 5f3c49a680439..4825b8b28e621 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -64,6 +64,7 @@ extern grammar _PyParser_Grammar; /* From graminit.c */ /* Forward declarations */ static PyStatus add_main_module(PyInterpreterState *interp); static PyStatus init_import_site(void); +static PyStatus init_set_builtins_open(PyThreadState *tstate); static PyStatus init_sys_streams(PyThreadState *tstate); static PyStatus init_signals(PyThreadState *tstate); static void call_py_exitfuncs(PyThreadState *tstate); @@ -994,6 +995,11 @@ pyinit_main(PyThreadState *tstate) return status; } + status = init_set_builtins_open(tstate); + if (_PyStatus_EXCEPTION(status)) { + return status; + } + /* Initialize warnings. */ PyObject *warnoptions = PySys_GetObject("warnoptions"); if (warnoptions != NULL && PyList_Size(warnoptions) > 0) @@ -1569,6 +1575,11 @@ new_interpreter(PyThreadState **tstate_p) return status; } + status = init_set_builtins_open(tstate); + if (_PyStatus_EXCEPTION(status)) { + return status; + } + status = add_main_module(interp); if (_PyStatus_EXCEPTION(status)) { return status; @@ -1891,12 +1902,49 @@ create_stdio(const PyConfig *config, PyObject* io, return NULL; } -/* Initialize sys.stdin, stdout, stderr and builtins.open */ +/* Set builtins.open to io.OpenWrapper */ static PyStatus -init_sys_streams(PyThreadState *tstate) +init_set_builtins_open(PyThreadState *tstate) { PyObject *iomod = NULL, *wrapper; PyObject *bimod = NULL; + PyStatus res = _PyStatus_OK(); + + if (!(iomod = PyImport_ImportModule("io"))) { + goto error; + } + + if (!(bimod = PyImport_ImportModule("builtins"))) { + goto error; + } + + if (!(wrapper = PyObject_GetAttrString(iomod, "OpenWrapper"))) { + goto error; + } + + /* Set builtins.open */ + if (PyObject_SetAttrString(bimod, "open", wrapper) == -1) { + Py_DECREF(wrapper); + goto error; + } + Py_DECREF(wrapper); + goto done; + +error: + res = _PyStatus_ERR("can't initialize io.open"); + +done: + Py_XDECREF(bimod); + Py_XDECREF(iomod); + return res; +} + + +/* Initialize sys.stdin, stdout, stderr and builtins.open */ +static PyStatus +init_sys_streams(PyThreadState *tstate) +{ + PyObject *iomod = NULL; PyObject *m; PyObject *std = NULL; int fd; @@ -1929,23 +1977,9 @@ init_sys_streams(PyThreadState *tstate) } Py_DECREF(m); - if (!(bimod = PyImport_ImportModule("builtins"))) { - goto error; - } - if (!(iomod = PyImport_ImportModule("io"))) { goto error; } - if (!(wrapper = PyObject_GetAttrString(iomod, "OpenWrapper"))) { - goto error; - } - - /* Set builtins.open */ - if (PyObject_SetAttrString(bimod, "open", wrapper) == -1) { - Py_DECREF(wrapper); - goto error; - } - Py_DECREF(wrapper); /* Set sys.stdin */ fd = fileno(stdin); @@ -2013,8 +2047,6 @@ init_sys_streams(PyThreadState *tstate) done: _Py_ClearStandardStreamEncoding(); - - Py_XDECREF(bimod); Py_XDECREF(iomod); return res; }

1 0

bpo-38804: Fix REDoS in http.cookiejar (GH-17157)
by Miss Islington (bot) 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/cb6085138a845f8324adc011b65754acc2… commit: cb6085138a845f8324adc011b65754acc2086cc0 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-11-22T06:42:13-08:00 summary: bpo-38804: Fix REDoS in http.cookiejar (GH-17157) The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular expression denial of service (REDoS). LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar to parse Set-Cookie headers returned by a server. Processing a response from a malicious HTTP server can lead to extreme CPU usage and execution will be blocked for a long time. The regex contained multiple overlapping \s* capture groups. Ignoring the ?-optional capture groups the regex could be simplified to \d+-\w+-\d+(\s*\s*\s*)$ Therefore, a long sequence of spaces can trigger bad performance. Matching a malicious string such as LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!") caused catastrophic backtracking. The fix removes ambiguity about which \s* should match a particular space. You can create a malicious server which responds with Set-Cookie headers to attack all python programs which access it e.g. from http.server import BaseHTTPRequestHandler, HTTPServer def make_set_cookie_value(n_spaces): spaces = " " * n_spaces expiry = f"1-c-1{spaces}!" return f"b;Expires={expiry}" class Handler(BaseHTTPRequestHandler): def do_GET(self): self.log_request(204) self.send_response_only(204) GH- Don't bother sending Server and Date n_spaces = ( int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences if len(self.path) > 1 else 65506 GH- Max header line length 65536 ) value = make_set_cookie_value(n_spaces) for i in range(99): GH- Not necessary, but we can have up to 100 header lines self.send_header("Set-Cookie", value) self.end_headers() if __name__ == "__main__": HTTPServer(("", 44020), Handler).serve_forever() This server returns 99 Set-Cookie headers. Each has 65506 spaces. Extracting the cookies will pretty much never complete. Vulnerable client using the example at the bottom of https://docs.python.org/3/library/http.cookiejar.html : import http.cookiejar, urllib.request cj = http.cookiejar.CookieJar() opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) r = opener.open("http://localhost:44020/") The popular requests library was also vulnerable without any additional options (as it uses http.cookiejar by default): import requests requests.get("http://localhost:44020/") * Regression test for http.cookiejar REDoS If we regress, this test will take a very long time. * Improve performance of http.cookiejar.ISO_DATE_RE A string like "444444" + (" " * 2000) + "A" could cause poor performance due to the 2 overlapping \s* groups, although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was. (cherry picked from commit 1b779bfb8593739b11cbb988ef82a883ec9d077e) Co-authored-by: bcaller <bcaller(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst M Lib/http/cookiejar.py M Lib/test/test_http_cookiejar.py M Misc/ACKS diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py index 1bed893b1e266..d43a2193f9128 100644 --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -213,10 +213,14 @@ def _str2time(day, mon, yr, hr, min, sec, tz): (?::(\d\d))? # optional seconds )? # optional clock \s* - ([-+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z]+)? # timezone + (?: + ([-+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z]+) # timezone + \s* + )? + (?: + $\w+$ # ASCII representation of timezone in parens. \s* - (?:$\w+$)? # ASCII representation of timezone in parens. - \s*$""", re.X | re.ASCII) + )?$""", re.X | re.ASCII) def http2time(text): """Returns time in seconds since epoch of time represented by a string. @@ -286,9 +290,11 @@ def http2time(text): (?::?(\d\d(?:\.\d*)?))? # optional seconds (and fractional) )? # optional clock \s* - ([-+]?\d\d?:?(:?\d\d)? - |Z|z)? # timezone (Z is "zero meridian", i.e. GMT) - \s*$""", re.X | re. ASCII) + (?: + ([-+]?\d\d?:?(:?\d\d)? + |Z|z) # timezone (Z is "zero meridian", i.e. GMT) + \s* + )?$""", re.X | re. ASCII) def iso2time(text): """ As for http2time, but parses the ISO 8601 formats: diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 6eabeeae999d8..ff86ffa98c3ee 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -123,6 +123,13 @@ def test_http2time_garbage(self): "http2time(%s) is not None\n" "http2time(test) %s" % (test, http2time(test))) + def test_http2time_redos_regression_actually_completes(self): + # LOOSE_HTTP_DATE_RE was vulnerable to malicious input which caused catastrophic backtracking (REDoS). + # If we regress to cubic complexity, this test will take a very long time to succeed. + # If fixed, it should complete within a fraction of a second. + http2time("01 Jan 1970{}00:00:00 GMT!".format(" " * 10 ** 5)) + http2time("01 Jan 1970 00:00:00{}GMT!".format(" " * 10 ** 5)) + def test_iso2time(self): def parse_date(text): return time.gmtime(iso2time(text))[:6] @@ -180,6 +187,12 @@ def test_iso2time_garbage(self): self.assertIsNone(iso2time(test), "iso2time(%r)" % test) + def test_iso2time_performance_regression(self): + # If ISO_DATE_RE regresses to quadratic complexity, this test will take a very long time to succeed. + # If fixed, it should complete within a fraction of a second. + iso2time('1994-02-03{}14:15:29 -0100!'.format(' '*10**6)) + iso2time('1994-02-03 14:15:29{}-0100!'.format(' '*10**6)) + class HeaderTests(unittest.TestCase): diff --git a/Misc/ACKS b/Misc/ACKS index 7afc92e7bb767..ad012eae7a323 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -245,6 +245,7 @@ Zach Byrne Vedran Čačić Nicolas Cadou Jp Calderone +Ben Caller Arnaud Calmettes Daniel Calvelo Tony Campbell diff --git a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst new file mode 100644 index 0000000000000..1f45142d9f743 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst @@ -0,0 +1 @@ +Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller.

1 0

bpo-38804: Fix REDoS in http.cookiejar (GH-17157)
by Miss Islington (bot) 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/a1e1be4c4969c7c20c8c958e5ab5279ae6… commit: a1e1be4c4969c7c20c8c958e5ab5279ae6a66a16 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-11-22T06:42:06-08:00 summary: bpo-38804: Fix REDoS in http.cookiejar (GH-17157) The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular expression denial of service (REDoS). LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar to parse Set-Cookie headers returned by a server. Processing a response from a malicious HTTP server can lead to extreme CPU usage and execution will be blocked for a long time. The regex contained multiple overlapping \s* capture groups. Ignoring the ?-optional capture groups the regex could be simplified to \d+-\w+-\d+(\s*\s*\s*)$ Therefore, a long sequence of spaces can trigger bad performance. Matching a malicious string such as LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!") caused catastrophic backtracking. The fix removes ambiguity about which \s* should match a particular space. You can create a malicious server which responds with Set-Cookie headers to attack all python programs which access it e.g. from http.server import BaseHTTPRequestHandler, HTTPServer def make_set_cookie_value(n_spaces): spaces = " " * n_spaces expiry = f"1-c-1{spaces}!" return f"b;Expires={expiry}" class Handler(BaseHTTPRequestHandler): def do_GET(self): self.log_request(204) self.send_response_only(204) GH- Don't bother sending Server and Date n_spaces = ( int(self.path[1:]) GH- Can GET e.g. /100 to test shorter sequences if len(self.path) > 1 else 65506 GH- Max header line length 65536 ) value = make_set_cookie_value(n_spaces) for i in range(99): GH- Not necessary, but we can have up to 100 header lines self.send_header("Set-Cookie", value) self.end_headers() if __name__ == "__main__": HTTPServer(("", 44020), Handler).serve_forever() This server returns 99 Set-Cookie headers. Each has 65506 spaces. Extracting the cookies will pretty much never complete. Vulnerable client using the example at the bottom of https://docs.python.org/3/library/http.cookiejar.html : import http.cookiejar, urllib.request cj = http.cookiejar.CookieJar() opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) r = opener.open("http://localhost:44020/") The popular requests library was also vulnerable without any additional options (as it uses http.cookiejar by default): import requests requests.get("http://localhost:44020/") * Regression test for http.cookiejar REDoS If we regress, this test will take a very long time. * Improve performance of http.cookiejar.ISO_DATE_RE A string like "444444" + (" " * 2000) + "A" could cause poor performance due to the 2 overlapping \s* groups, although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was. (cherry picked from commit 1b779bfb8593739b11cbb988ef82a883ec9d077e) Co-authored-by: bcaller <bcaller(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst M Lib/http/cookiejar.py M Lib/test/test_http_cookiejar.py M Misc/ACKS diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py index adc7ed62425d0..47ed5c3d64ab7 100644 --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -214,10 +214,14 @@ def _str2time(day, mon, yr, hr, min, sec, tz): (?::(\d\d))? # optional seconds )? # optional clock \s* - ([-+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z]+)? # timezone + (?: + ([-+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z]+) # timezone + \s* + )? + (?: + $\w+$ # ASCII representation of timezone in parens. \s* - (?:$\w+$)? # ASCII representation of timezone in parens. - \s*$""", re.X | re.ASCII) + )?$""", re.X | re.ASCII) def http2time(text): """Returns time in seconds since epoch of time represented by a string. @@ -287,9 +291,11 @@ def http2time(text): (?::?(\d\d(?:\.\d*)?))? # optional seconds (and fractional) )? # optional clock \s* - ([-+]?\d\d?:?(:?\d\d)? - |Z|z)? # timezone (Z is "zero meridian", i.e. GMT) - \s*$""", re.X | re. ASCII) + (?: + ([-+]?\d\d?:?(:?\d\d)? + |Z|z) # timezone (Z is "zero meridian", i.e. GMT) + \s* + )?$""", re.X | re. ASCII) def iso2time(text): """ As for http2time, but parses the ISO 8601 formats: diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 853a40044965b..2d7077af6da39 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -123,6 +123,13 @@ def test_http2time_garbage(self): "http2time(%s) is not None\n" "http2time(test) %s" % (test, http2time(test))) + def test_http2time_redos_regression_actually_completes(self): + # LOOSE_HTTP_DATE_RE was vulnerable to malicious input which caused catastrophic backtracking (REDoS). + # If we regress to cubic complexity, this test will take a very long time to succeed. + # If fixed, it should complete within a fraction of a second. + http2time("01 Jan 1970{}00:00:00 GMT!".format(" " * 10 ** 5)) + http2time("01 Jan 1970 00:00:00{}GMT!".format(" " * 10 ** 5)) + def test_iso2time(self): def parse_date(text): return time.gmtime(iso2time(text))[:6] @@ -180,6 +187,12 @@ def test_iso2time_garbage(self): self.assertIsNone(iso2time(test), "iso2time(%r)" % test) + def test_iso2time_performance_regression(self): + # If ISO_DATE_RE regresses to quadratic complexity, this test will take a very long time to succeed. + # If fixed, it should complete within a fraction of a second. + iso2time('1994-02-03{}14:15:29 -0100!'.format(' '*10**6)) + iso2time('1994-02-03 14:15:29{}-0100!'.format(' '*10**6)) + class HeaderTests(unittest.TestCase): diff --git a/Misc/ACKS b/Misc/ACKS index a2e8046313f0a..e52ae984f1571 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -250,6 +250,7 @@ Zach Byrne Vedran Čačić Nicolas Cadou Jp Calderone +Ben Caller Arnaud Calmettes Daniel Calvelo Tony Campbell diff --git a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst new file mode 100644 index 0000000000000..1f45142d9f743 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst @@ -0,0 +1 @@ +Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller.

1 0

bpo-38804: Fix REDoS in http.cookiejar (GH-17157)
by Victor Stinner 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/1b779bfb8593739b11cbb988ef82a883ec… commit: 1b779bfb8593739b11cbb988ef82a883ec9d077e branch: master author: bcaller <bcaller(a)users.noreply.github.com> committer: Victor Stinner <vstinner(a)python.org> date: 2019-11-22T15:22:11+01:00 summary: bpo-38804: Fix REDoS in http.cookiejar (GH-17157) The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular expression denial of service (REDoS). LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar to parse Set-Cookie headers returned by a server. Processing a response from a malicious HTTP server can lead to extreme CPU usage and execution will be blocked for a long time. The regex contained multiple overlapping \s* capture groups. Ignoring the ?-optional capture groups the regex could be simplified to \d+-\w+-\d+(\s*\s*\s*)$ Therefore, a long sequence of spaces can trigger bad performance. Matching a malicious string such as LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!") caused catastrophic backtracking. The fix removes ambiguity about which \s* should match a particular space. You can create a malicious server which responds with Set-Cookie headers to attack all python programs which access it e.g. from http.server import BaseHTTPRequestHandler, HTTPServer def make_set_cookie_value(n_spaces): spaces = " " * n_spaces expiry = f"1-c-1{spaces}!" return f"b;Expires={expiry}" class Handler(BaseHTTPRequestHandler): def do_GET(self): self.log_request(204) self.send_response_only(204) # Don't bother sending Server and Date n_spaces = ( int(self.path[1:]) # Can GET e.g. /100 to test shorter sequences if len(self.path) > 1 else 65506 # Max header line length 65536 ) value = make_set_cookie_value(n_spaces) for i in range(99): # Not necessary, but we can have up to 100 header lines self.send_header("Set-Cookie", value) self.end_headers() if __name__ == "__main__": HTTPServer(("", 44020), Handler).serve_forever() This server returns 99 Set-Cookie headers. Each has 65506 spaces. Extracting the cookies will pretty much never complete. Vulnerable client using the example at the bottom of https://docs.python.org/3/library/http.cookiejar.html : import http.cookiejar, urllib.request cj = http.cookiejar.CookieJar() opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj)) r = opener.open("http://localhost:44020/") The popular requests library was also vulnerable without any additional options (as it uses http.cookiejar by default): import requests requests.get("http://localhost:44020/") * Regression test for http.cookiejar REDoS If we regress, this test will take a very long time. * Improve performance of http.cookiejar.ISO_DATE_RE A string like "444444" + (" " * 2000) + "A" could cause poor performance due to the 2 overlapping \s* groups, although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was. files: A Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst M Lib/http/cookiejar.py M Lib/test/test_http_cookiejar.py M Misc/ACKS diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py index adc7ed62425d0..47ed5c3d64ab7 100644 --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -214,10 +214,14 @@ def _str2time(day, mon, yr, hr, min, sec, tz): (?::(\d\d))? # optional seconds )? # optional clock \s* - ([-+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z]+)? # timezone + (?: + ([-+]?\d{2,4}|(?![APap][Mm]\b)[A-Za-z]+) # timezone + \s* + )? + (?: + $\w+$ # ASCII representation of timezone in parens. \s* - (?:$\w+$)? # ASCII representation of timezone in parens. - \s*$""", re.X | re.ASCII) + )?$""", re.X | re.ASCII) def http2time(text): """Returns time in seconds since epoch of time represented by a string. @@ -287,9 +291,11 @@ def http2time(text): (?::?(\d\d(?:\.\d*)?))? # optional seconds (and fractional) )? # optional clock \s* - ([-+]?\d\d?:?(:?\d\d)? - |Z|z)? # timezone (Z is "zero meridian", i.e. GMT) - \s*$""", re.X | re. ASCII) + (?: + ([-+]?\d\d?:?(:?\d\d)? + |Z|z) # timezone (Z is "zero meridian", i.e. GMT) + \s* + )?$""", re.X | re. ASCII) def iso2time(text): """ As for http2time, but parses the ISO 8601 formats: diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 853a40044965b..2d7077af6da39 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -123,6 +123,13 @@ def test_http2time_garbage(self): "http2time(%s) is not None\n" "http2time(test) %s" % (test, http2time(test))) + def test_http2time_redos_regression_actually_completes(self): + # LOOSE_HTTP_DATE_RE was vulnerable to malicious input which caused catastrophic backtracking (REDoS). + # If we regress to cubic complexity, this test will take a very long time to succeed. + # If fixed, it should complete within a fraction of a second. + http2time("01 Jan 1970{}00:00:00 GMT!".format(" " * 10 ** 5)) + http2time("01 Jan 1970 00:00:00{}GMT!".format(" " * 10 ** 5)) + def test_iso2time(self): def parse_date(text): return time.gmtime(iso2time(text))[:6] @@ -180,6 +187,12 @@ def test_iso2time_garbage(self): self.assertIsNone(iso2time(test), "iso2time(%r)" % test) + def test_iso2time_performance_regression(self): + # If ISO_DATE_RE regresses to quadratic complexity, this test will take a very long time to succeed. + # If fixed, it should complete within a fraction of a second. + iso2time('1994-02-03{}14:15:29 -0100!'.format(' '*10**6)) + iso2time('1994-02-03 14:15:29{}-0100!'.format(' '*10**6)) + class HeaderTests(unittest.TestCase): diff --git a/Misc/ACKS b/Misc/ACKS index 13c6676bace1f..357ce024e9e8c 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -250,6 +250,7 @@ Zach Byrne Vedran Čačić Nicolas Cadou Jp Calderone +Ben Caller Arnaud Calmettes Daniel Calvelo Tony Campbell diff --git a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst new file mode 100644 index 0000000000000..1f45142d9f743 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst @@ -0,0 +1 @@ +Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller.

1 0

bpo-22367: Update test_fcntl.py for spawn process mode (GH-17154) (GH-17253)
by Victor Stinner 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/d4d79209e69d52ea4c047545c6c60c3ba7… commit: d4d79209e69d52ea4c047545c6c60c3ba75f15f4 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Victor Stinner <vstinner(a)python.org> date: 2019-11-22T15:15:42+01:00 summary: bpo-22367: Update test_fcntl.py for spawn process mode (GH-17154) (GH-17253) (cherry picked from commit 9960230f76eb555d6dfbe8a324efed35610c85f9) Co-authored-by: Dong-hee Na <donghee.na92(a)gmail.com> files: M Lib/test/test_fcntl.py diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py index 30eeb6dc4d02a..036bd64b40511 100644 --- a/Lib/test/test_fcntl.py +++ b/Lib/test/test_fcntl.py @@ -51,6 +51,21 @@ def __init__(self, fn): def fileno(self): return self.fn +def try_lockf_on_other_process_fail(fname, cmd): + f = open(fname, 'wb+') + try: + fcntl.lockf(f, cmd) + except BlockingIOError: + pass + finally: + f.close() + +def try_lockf_on_other_process(fname, cmd): + f = open(fname, 'wb+') + fcntl.lockf(f, cmd) + fcntl.lockf(f, fcntl.LOCK_UN) + f.close() + class TestFcntl(unittest.TestCase): def setUp(self): @@ -138,28 +153,23 @@ def test_flock(self): self.assertRaises(ValueError, fcntl.flock, -1, fcntl.LOCK_SH) self.assertRaises(TypeError, fcntl.flock, 'spam', fcntl.LOCK_SH) + @unittest.skipIf(platform.system() == "AIX", "AIX returns PermissionError") def test_lockf_exclusive(self): self.f = open(TESTFN, 'wb+') cmd = fcntl.LOCK_EX | fcntl.LOCK_NB - def try_lockf_on_other_process(): - self.assertRaises(BlockingIOError, fcntl.lockf, self.f, cmd) - fcntl.lockf(self.f, cmd) - p = Process(target=try_lockf_on_other_process) + p = Process(target=try_lockf_on_other_process_fail, args=(TESTFN, cmd)) p.start() p.join() fcntl.lockf(self.f, fcntl.LOCK_UN) self.assertEqual(p.exitcode, 0) + @unittest.skipIf(platform.system() == "AIX", "AIX returns PermissionError") def test_lockf_share(self): self.f = open(TESTFN, 'wb+') cmd = fcntl.LOCK_SH | fcntl.LOCK_NB - def try_lockf_on_other_process(): - fcntl.lockf(self.f, cmd) - fcntl.lockf(self.f, fcntl.LOCK_UN) - fcntl.lockf(self.f, cmd) - p = Process(target=try_lockf_on_other_process) + p = Process(target=try_lockf_on_other_process, args=(TESTFN, cmd)) p.start() p.join() fcntl.lockf(self.f, fcntl.LOCK_UN)

1 0

bpo-22367: Update test_fcntl.py for spawn process mode (GH-17154) (GH-17252)
by Victor Stinner 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/c3cd0de9ec7ab54186cebef5b2edfd098f… commit: c3cd0de9ec7ab54186cebef5b2edfd098f7ae387 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Victor Stinner <vstinner(a)python.org> date: 2019-11-22T15:15:36+01:00 summary: bpo-22367: Update test_fcntl.py for spawn process mode (GH-17154) (GH-17252) (cherry picked from commit 9960230f76eb555d6dfbe8a324efed35610c85f9) Co-authored-by: Dong-hee Na <donghee.na92(a)gmail.com> files: M Lib/test/test_fcntl.py diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py index 88d989d5e358d..c394f6945a8d2 100644 --- a/Lib/test/test_fcntl.py +++ b/Lib/test/test_fcntl.py @@ -51,6 +51,21 @@ def __init__(self, fn): def fileno(self): return self.fn +def try_lockf_on_other_process_fail(fname, cmd): + f = open(fname, 'wb+') + try: + fcntl.lockf(f, cmd) + except BlockingIOError: + pass + finally: + f.close() + +def try_lockf_on_other_process(fname, cmd): + f = open(fname, 'wb+') + fcntl.lockf(f, cmd) + fcntl.lockf(f, fcntl.LOCK_UN) + f.close() + class TestFcntl(unittest.TestCase): def setUp(self): @@ -138,28 +153,23 @@ def test_flock(self): self.assertRaises(ValueError, fcntl.flock, -1, fcntl.LOCK_SH) self.assertRaises(TypeError, fcntl.flock, 'spam', fcntl.LOCK_SH) + @unittest.skipIf(platform.system() == "AIX", "AIX returns PermissionError") def test_lockf_exclusive(self): self.f = open(TESTFN, 'wb+') cmd = fcntl.LOCK_EX | fcntl.LOCK_NB - def try_lockf_on_other_process(): - self.assertRaises(BlockingIOError, fcntl.lockf, self.f, cmd) - fcntl.lockf(self.f, cmd) - p = Process(target=try_lockf_on_other_process) + p = Process(target=try_lockf_on_other_process_fail, args=(TESTFN, cmd)) p.start() p.join() fcntl.lockf(self.f, fcntl.LOCK_UN) self.assertEqual(p.exitcode, 0) + @unittest.skipIf(platform.system() == "AIX", "AIX returns PermissionError") def test_lockf_share(self): self.f = open(TESTFN, 'wb+') cmd = fcntl.LOCK_SH | fcntl.LOCK_NB - def try_lockf_on_other_process(): - fcntl.lockf(self.f, cmd) - fcntl.lockf(self.f, fcntl.LOCK_UN) - fcntl.lockf(self.f, cmd) - p = Process(target=try_lockf_on_other_process) + p = Process(target=try_lockf_on_other_process, args=(TESTFN, cmd)) p.start() p.join() fcntl.lockf(self.f, fcntl.LOCK_UN)

1 0

Fix quoted signature of setattrofunc (GH-17251)
by Victor Stinner 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/4e205b74f91400c0e6810e79370a99e29a… commit: 4e205b74f91400c0e6810e79370a99e29a28d599 branch: master author: Alex <637714+alexchandel(a)users.noreply.github.com> committer: Victor Stinner <vstinner(a)python.org> date: 2019-11-22T14:48:14+01:00 summary: Fix quoted signature of setattrofunc (GH-17251) setattrofunc returns `int`, not `PyObject *`. files: M Doc/c-api/typeobj.rst diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index b4ffd6b41c628..7b205c044953b 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -955,7 +955,7 @@ and :c:type:`PyType_Type` effectively act as defaults.) The signature is the same as for :c:func:`PyObject_SetAttr`:: - PyObject *tp_setattro(PyObject *self, PyObject *attr, PyObject *value); + int tp_setattro(PyObject *self, PyObject *attr, PyObject *value); In addition, setting *value* to ``NULL`` to delete an attribute must be supported. It is usually convenient to set this field to

1 0

bpo-36854: Fix reference counter in PyInit__testcapi() (GH-17338)
by Miss Islington (bot) 22 Nov '19

22 Nov '19

https://github.com/python/cpython/commit/bff525566469021949910deec84e837306… commit: bff525566469021949910deec84e837306b79886 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-11-22T04:56:59-08:00 summary: bpo-36854: Fix reference counter in PyInit__testcapi() (GH-17338) Increment properly Py_True/Py_False reference counter for _testcapi.WITH_PYMALLOC variable. (cherry picked from commit 84c36c152a2bdf98f9cc7ce0e1db98e1f442a05e) Co-authored-by: Victor Stinner <vstinner(a)python.org> files: M Modules/_testcapimodule.c diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 64b7b69049394..16ea9c2d66554 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5444,11 +5444,14 @@ PyInit__testcapi(void) PyModule_AddObject(m, "instancemethod", (PyObject *)&PyInstanceMethod_Type); PyModule_AddIntConstant(m, "the_number_three", 3); + PyObject *v; #ifdef WITH_PYMALLOC - PyModule_AddObject(m, "WITH_PYMALLOC", Py_True); + v = Py_True; #else - PyModule_AddObject(m, "WITH_PYMALLOC", Py_False); + v = Py_False; #endif + Py_INCREF(v); + PyModule_AddObject(m, "WITH_PYMALLOC", v); TestError = PyErr_NewException("_testcapi.error", NULL, NULL); Py_INCREF(TestError);

1 0