Mailman 3 July 2020 - Python-checkins

bpo-32528: Document the change in inheritance of asyncio.CancelledError (GH-21474)
by Miss Islington (bot) 19 Jul '20

19 Jul '20

https://github.com/python/cpython/commit/700cb6617545cdb8a9e16bb2e6efe90788… commit: 700cb6617545cdb8a9e16bb2e6efe90788a60d4d branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-19T00:50:54-07:00 summary: bpo-32528: Document the change in inheritance of asyncio.CancelledError (GH-21474) GH-msg373510 [bpo-32528]()/GH-13528 changed `asyncio.CancelledError` such that it no longer inherits from `concurrent.futures.CancelledError`. As this affects existing code, specifically when catching the latter instead of the former in exception handling, it should be documented in the "What's new in 3.8?" document. Automerge-Triggered-By: @1st1 (cherry picked from commit 2a5181829af394b82e8e8c917183c709ee72a2b7) Co-authored-by: JustAnotherArchivist <JustAnotherArchivist(a)users.noreply.github.com> files: M Doc/whatsnew/3.8.rst diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index d2db5bff5afcf..475e3ab964312 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -648,7 +648,8 @@ loop on every invocation: (Contributed by Yury Selivanov in :issue:`37028`.) The exception :class:`asyncio.CancelledError` now inherits from -:class:`BaseException` rather than :class:`Exception`. +:class:`BaseException` rather than :class:`Exception` and no longer inherits +from :class:`concurrent.futures.CancelledError`. (Contributed by Yury Selivanov in :issue:`32528`.) On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`. @@ -1953,7 +1954,8 @@ Changes in the Python API (Contributed by Anthony Sottile in :issue:`36264`.) * The exception :class:`asyncio.CancelledError` now inherits from - :class:`BaseException` rather than :class:`Exception`. + :class:`BaseException` rather than :class:`Exception` and no longer inherits + from :class:`concurrent.futures.CancelledError`. (Contributed by Yury Selivanov in :issue:`32528`.) * The function :func:`asyncio.wait_for` now correctly waits for cancellation

1 0

bpo-41333: Convert OrderedDict.pop() to Argument Clinic (GH-21534)
by Serhiy Storchaka 19 Jul '20

19 Jul '20

https://github.com/python/cpython/commit/6bf3237379b17632db52cb39d181e8bac7… commit: 6bf3237379b17632db52cb39d181e8bac70173f3 branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-07-19T09:18:55+03:00 summary: bpo-41333: Convert OrderedDict.pop() to Argument Clinic (GH-21534) files: A Misc/NEWS.d/next/Library/2020-07-18-18-07-40.bpo-41333.upkHIm.rst M Objects/clinic/dictobject.c.h M Objects/clinic/odictobject.c.h M Objects/dictobject.c M Objects/odictobject.c diff --git a/Misc/NEWS.d/next/Library/2020-07-18-18-07-40.bpo-41333.upkHIm.rst b/Misc/NEWS.d/next/Library/2020-07-18-18-07-40.bpo-41333.upkHIm.rst new file mode 100644 index 0000000000000..73e8b1199772d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-07-18-18-07-40.bpo-41333.upkHIm.rst @@ -0,0 +1 @@ +:meth:`collections.OrderedDict.pop` is now 2 times faster. diff --git a/Objects/clinic/dictobject.c.h b/Objects/clinic/dictobject.c.h index 7395e3bcebba8..beb3f360f8634 100644 --- a/Objects/clinic/dictobject.c.h +++ b/Objects/clinic/dictobject.c.h @@ -122,7 +122,8 @@ PyDoc_STRVAR(dict_pop__doc__, "\n" "D.pop(k[,d]) -> v, remove specified key and return the corresponding value.\n" "\n" -"If key is not found, default is returned if given, otherwise KeyError is raised"); +"If the key is not found, return the default if given; otherwise,\n" +"raise a KeyError."); #define DICT_POP_METHODDEF \ {"pop", (PyCFunction)(void(*)(void))dict_pop, METH_FASTCALL, dict_pop__doc__}, @@ -190,4 +191,4 @@ dict___reversed__(PyDictObject *self, PyObject *Py_UNUSED(ignored)) { return dict___reversed___impl(self); } -/*[clinic end generated code: output=4d98145508da8fa3 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=7b77c16e43d6735a input=a9049054013a1b77]*/ diff --git a/Objects/clinic/odictobject.c.h b/Objects/clinic/odictobject.c.h index f43bc14ce1b62..a3ab9ea507f44 100644 --- a/Objects/clinic/odictobject.c.h +++ b/Objects/clinic/odictobject.c.h @@ -83,6 +83,49 @@ OrderedDict_setdefault(PyODictObject *self, PyObject *const *args, Py_ssize_t na return return_value; } +PyDoc_STRVAR(OrderedDict_pop__doc__, +"pop($self, /, key, default=<unrepresentable>)\n" +"--\n" +"\n" +"od.pop(key[,default]) -> v, remove specified key and return the corresponding value.\n" +"\n" +"If the key is not found, return the default if given; otherwise,\n" +"raise a KeyError."); + +#define ORDEREDDICT_POP_METHODDEF \ + {"pop", (PyCFunction)(void(*)(void))OrderedDict_pop, METH_FASTCALL|METH_KEYWORDS, OrderedDict_pop__doc__}, + +static PyObject * +OrderedDict_pop_impl(PyODictObject *self, PyObject *key, + PyObject *default_value); + +static PyObject * +OrderedDict_pop(PyODictObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"key", "default", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "pop", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; + PyObject *key; + PyObject *default_value = NULL; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 2, 0, argsbuf); + if (!args) { + goto exit; + } + key = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + default_value = args[1]; +skip_optional_pos: + return_value = OrderedDict_pop_impl(self, key, default_value); + +exit: + return return_value; +} + PyDoc_STRVAR(OrderedDict_popitem__doc__, "popitem($self, /, last=True)\n" "--\n" @@ -168,4 +211,4 @@ OrderedDict_move_to_end(PyODictObject *self, PyObject *const *args, Py_ssize_t n exit: return return_value; } -/*[clinic end generated code: output=8eb1296df9142908 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e0afaad5b4bb47fe input=a9049054013a1b77]*/ diff --git a/Objects/dictobject.c b/Objects/dictobject.c index b1f11b3e695bb..ba22489539ae7 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -3054,12 +3054,13 @@ dict.pop D.pop(k[,d]) -> v, remove specified key and return the corresponding value. -If key is not found, default is returned if given, otherwise KeyError is raised +If the key is not found, return the default if given; otherwise, +raise a KeyError. [clinic start generated code]*/ static PyObject * dict_pop_impl(PyDictObject *self, PyObject *key, PyObject *default_value) -/*[clinic end generated code: output=3abb47b89f24c21c input=eeebec7812190348]*/ +/*[clinic end generated code: output=3abb47b89f24c21c input=e221baa01044c44c]*/ { return _PyDict_Pop((PyObject*)self, key, default_value); } diff --git a/Objects/odictobject.c b/Objects/odictobject.c index d5bf499575d09..b4ac560d23581 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -1045,30 +1045,28 @@ OrderedDict_setdefault_impl(PyODictObject *self, PyObject *key, /* pop() */ -PyDoc_STRVAR(odict_pop__doc__, -"od.pop(k[,d]) -> v, remove specified key and return the corresponding\n\ - value. If key is not found, d is returned if given, otherwise KeyError\n\ - is raised.\n\ -\n\ - "); - /* forward */ static PyObject * _odict_popkey(PyObject *, PyObject *, PyObject *); /* Skips __missing__() calls. */ -static PyObject * -odict_pop(PyObject *od, PyObject *args, PyObject *kwargs) -{ - static char *kwlist[] = {"key", "default", 0}; - PyObject *key, *failobj = NULL; +/*[clinic input] +OrderedDict.pop - /* borrowed */ - if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|O:pop", kwlist, - &key, &failobj)) { - return NULL; - } + key: object + default: object = NULL + +od.pop(key[,default]) -> v, remove specified key and return the corresponding value. + +If the key is not found, return the default if given; otherwise, +raise a KeyError. +[clinic start generated code]*/ - return _odict_popkey(od, key, failobj); +static PyObject * +OrderedDict_pop_impl(PyODictObject *self, PyObject *key, + PyObject *default_value) +/*[clinic end generated code: output=7a6447d104e7494b input=7efe36601007dff7]*/ +{ + return _odict_popkey((PyObject *)self, key, default_value); } static PyObject * @@ -1362,8 +1360,7 @@ static PyMethodDef odict_methods[] = { {"__reduce__", (PyCFunction)odict_reduce, METH_NOARGS, odict_reduce__doc__}, ORDEREDDICT_SETDEFAULT_METHODDEF - {"pop", (PyCFunction)(void(*)(void))odict_pop, - METH_VARARGS | METH_KEYWORDS, odict_pop__doc__}, + ORDEREDDICT_POP_METHODDEF ORDEREDDICT_POPITEM_METHODDEF {"keys", odictkeys_new, METH_NOARGS, odict_keys__doc__},

1 0

bpo-41295: Reimplement the Carlo Verre "hackcheck" (GH-21528)
by Miss Islington (bot) 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/38d930f2ccbff6f93c4c54a7a6a1759266… commit: 38d930f2ccbff6f93c4c54a7a6a1759266136504 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T14:37:43-07:00 summary: bpo-41295: Reimplement the Carlo Verre "hackcheck" (GH-21528) Walk down the MRO backwards to find the type that originally defined the final `tp_setattro`, then make sure we are not jumping over intermediate C-level bases with the Python-level call. Automerge-Triggered-By: @gvanrossum (cherry picked from commit c53b310e5926266ce267c44a168165cacd786d6e) Co-authored-by: scoder <stefan_ml(a)behnel.de> files: A Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst M Lib/test/test_descr.py M Objects/typeobject.c diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index c4d900a8c1bcb..18397a86b14c5 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4315,6 +4315,42 @@ def test_carloverre(self): else: self.fail("Carlo Verre __delattr__ succeeded!") + def test_carloverre_multi_inherit_valid(self): + class A(type): + def __setattr__(cls, key, value): + type.__setattr__(cls, key, value) + + class B: + pass + + class C(B, A): + pass + + obj = C('D', (object,), {}) + try: + obj.test = True + except TypeError: + self.fail("setattr through direct base types should be legal") + + def test_carloverre_multi_inherit_invalid(self): + class A(type): + def __setattr__(cls, key, value): + object.__setattr__(cls, key, value) # this should fail! + + class B: + pass + + class C(B, A): + pass + + obj = C('D', (object,), {}) + try: + obj.test = True + except TypeError: + pass + else: + self.fail("setattr through indirect base types should be rejected") + def test_weakref_segfault(self): # Testing weakref segfault... # SF 742911 diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst b/Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst new file mode 100644 index 0000000000000..d61fd8f0a2968 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst @@ -0,0 +1,3 @@ +Resolve a regression in CPython 3.8.4 where defining "__setattr__" in a +multi-inheritance setup and calling up the hierarchy chain could fail +if builtins/extension types were involved in the base types. diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 00128698cd018..0acde714a87b1 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -5822,14 +5822,29 @@ hackcheck(PyObject *self, setattrofunc func, const char *what) return 1; } assert(PyTuple_Check(mro)); - Py_ssize_t i, n; - n = PyTuple_GET_SIZE(mro); - for (i = 0; i < n; i++) { + + /* Find the (base) type that defined the type's slot function. */ + PyTypeObject *defining_type = type; + Py_ssize_t i; + for (i = PyTuple_GET_SIZE(mro) - 1; i >= 0; i--) { PyTypeObject *base = (PyTypeObject*) PyTuple_GET_ITEM(mro, i); + if (base->tp_setattro == slot_tp_setattro) { + /* Ignore Python classes: + they never define their own C-level setattro. */ + } + else if (base->tp_setattro == type->tp_setattro) { + defining_type = base; + break; + } + } + + /* Reject calls that jump over intermediate C-level overrides. */ + for (PyTypeObject *base = defining_type; base; base = base->tp_base) { if (base->tp_setattro == func) { - /* 'func' is the earliest non-Python implementation in the MRO. */ + /* 'func' is the right slot function to call. */ break; - } else if (base->tp_setattro != slot_tp_setattro) { + } + else if (base->tp_setattro != slot_tp_setattro) { /* 'base' is not a Python class and overrides 'func'. Its tp_setattro should be called instead. */ PyErr_Format(PyExc_TypeError, @@ -5839,8 +5854,6 @@ hackcheck(PyObject *self, setattrofunc func, const char *what) return 0; } } - /* Either 'func' is not in the mro (which should fail when checking 'self'), - or it's the right slot function to call. */ return 1; }

1 0

bpo-41295: Reimplement the Carlo Verre "hackcheck" (GH-21528)
by scoder 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/c53b310e5926266ce267c44a168165cacd… commit: c53b310e5926266ce267c44a168165cacd786d6e branch: master author: scoder <stefan_ml(a)behnel.de> committer: GitHub <noreply(a)github.com> date: 2020-07-18T14:19:50-07:00 summary: bpo-41295: Reimplement the Carlo Verre "hackcheck" (GH-21528) Walk down the MRO backwards to find the type that originally defined the final `tp_setattro`, then make sure we are not jumping over intermediate C-level bases with the Python-level call. Automerge-Triggered-By: @gvanrossum files: A Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst M Lib/test/test_descr.py M Objects/typeobject.c diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 7bb6f2bb4b30b..9738fb52a04bf 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4308,6 +4308,42 @@ def test_carloverre(self): else: self.fail("Carlo Verre __delattr__ succeeded!") + def test_carloverre_multi_inherit_valid(self): + class A(type): + def __setattr__(cls, key, value): + type.__setattr__(cls, key, value) + + class B: + pass + + class C(B, A): + pass + + obj = C('D', (object,), {}) + try: + obj.test = True + except TypeError: + self.fail("setattr through direct base types should be legal") + + def test_carloverre_multi_inherit_invalid(self): + class A(type): + def __setattr__(cls, key, value): + object.__setattr__(cls, key, value) # this should fail! + + class B: + pass + + class C(B, A): + pass + + obj = C('D', (object,), {}) + try: + obj.test = True + except TypeError: + pass + else: + self.fail("setattr through indirect base types should be rejected") + def test_weakref_segfault(self): # Testing weakref segfault... # SF 742911 diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst b/Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst new file mode 100644 index 0000000000000..d61fd8f0a2968 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-07-18-08-15-32.bpo-41295.pu8Ezo.rst @@ -0,0 +1,3 @@ +Resolve a regression in CPython 3.8.4 where defining "__setattr__" in a +multi-inheritance setup and calling up the hierarchy chain could fail +if builtins/extension types were involved in the base types. diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 16f95f0e1bf7f..49b3c859e35c1 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -5961,14 +5961,29 @@ hackcheck(PyObject *self, setattrofunc func, const char *what) return 1; } assert(PyTuple_Check(mro)); - Py_ssize_t i, n; - n = PyTuple_GET_SIZE(mro); - for (i = 0; i < n; i++) { + + /* Find the (base) type that defined the type's slot function. */ + PyTypeObject *defining_type = type; + Py_ssize_t i; + for (i = PyTuple_GET_SIZE(mro) - 1; i >= 0; i--) { PyTypeObject *base = (PyTypeObject*) PyTuple_GET_ITEM(mro, i); + if (base->tp_setattro == slot_tp_setattro) { + /* Ignore Python classes: + they never define their own C-level setattro. */ + } + else if (base->tp_setattro == type->tp_setattro) { + defining_type = base; + break; + } + } + + /* Reject calls that jump over intermediate C-level overrides. */ + for (PyTypeObject *base = defining_type; base; base = base->tp_base) { if (base->tp_setattro == func) { - /* 'func' is the earliest non-Python implementation in the MRO. */ + /* 'func' is the right slot function to call. */ break; - } else if (base->tp_setattro != slot_tp_setattro) { + } + else if (base->tp_setattro != slot_tp_setattro) { /* 'base' is not a Python class and overrides 'func'. Its tp_setattro should be called instead. */ PyErr_Format(PyExc_TypeError, @@ -5978,8 +5993,6 @@ hackcheck(PyObject *self, setattrofunc func, const char *what) return 0; } } - /* Either 'func' is not in the mro (which should fail when checking 'self'), - or it's the right slot function to call. */ return 1; }

1 0

bpo-39603: Prevent header injection in http methods (GH-18485)
by Miss Islington (bot) 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/668d321476d974c4f51476b33aaca87027… commit: 668d321476d974c4f51476b33aaca870272523bf branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T13:39:12-07:00 summary: bpo-39603: Prevent header injection in http methods (GH-18485) reject control chars in http method in http.client.putrequest to prevent http header injection (cherry picked from commit 8ca8a2e8fb068863c1138f07e3098478ef8be12e) Co-authored-by: AMIR <31338382+amiremohamadi(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst M Lib/http/client.py M Lib/test/test_httplib.py diff --git a/Lib/http/client.py b/Lib/http/client.py index 019380a720318..c2ad0471bfee5 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -147,6 +147,10 @@ # _is_allowed_url_pchars_re = re.compile(r"^[/!$&'()*+,;=:@%a-zA-Z0-9._~-]+$") # We are more lenient for assumed real world compatibility purposes. +# These characters are not allowed within HTTP method names +# to prevent http header injection. +_contains_disallowed_method_pchar_re = re.compile('[\x00-\x1f]') + # We always set the Content-Length header for these methods because some # servers will otherwise respond with a 411 _METHODS_EXPECTING_BODY = {'PATCH', 'POST', 'PUT'} @@ -1087,6 +1091,8 @@ def putrequest(self, method, url, skip_host=False, else: raise CannotSendRequest(self.__state) + self._validate_method(method) + # Save the method for use later in the response phase self._method = method @@ -1177,6 +1183,15 @@ def _encode_request(self, request): # ASCII also helps prevent CVE-2019-9740. return request.encode('ascii') + def _validate_method(self, method): + """Validate a method name for putrequest.""" + # prevent http header injection + match = _contains_disallowed_method_pchar_re.search(method) + if match: + raise ValueError( + f"method can't contain control characters. {method!r} " + f"(found at least {match.group()!r})") + def _validate_path(self, url): """Validate a url for putrequest.""" # Prevent CVE-2019-9740. diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py index 8f0e27a1fb836..5a5fcecbc9c15 100644 --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -364,6 +364,28 @@ def test_headers_debuglevel(self): self.assertEqual(lines[3], "header: Second: val2") +class HttpMethodTests(TestCase): + def test_invalid_method_names(self): + methods = ( + 'GET\r', + 'POST\n', + 'PUT\n\r', + 'POST\nValue', + 'POST\nHOST:abc', + 'GET\nrHost:abc\n', + 'POST\rRemainder:\r', + 'GET\rHOST:\n', + '\nPUT' + ) + + for method in methods: + with self.assertRaisesRegex( + ValueError, "method can't contain control characters"): + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn.request(method=method, url="/") + + class TransferEncodingTest(TestCase): expected_body = b"It's just a flesh wound" diff --git a/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst new file mode 100644 index 0000000000000..990affc3edd9d --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst @@ -0,0 +1,2 @@ +Prevent http header injection by rejecting control characters in +http.client.putrequest(...).

1 0

bpo-39603: Prevent header injection in http methods (GH-18485)
by AMIR 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/8ca8a2e8fb068863c1138f07e3098478ef… commit: 8ca8a2e8fb068863c1138f07e3098478ef8be12e branch: master author: AMIR <31338382+amiremohamadi(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T13:16:10-07:00 summary: bpo-39603: Prevent header injection in http methods (GH-18485) reject control chars in http method in http.client.putrequest to prevent http header injection files: A Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst M Lib/http/client.py M Lib/test/test_httplib.py diff --git a/Lib/http/client.py b/Lib/http/client.py index 500230d5d514f..15abcfeada591 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -147,6 +147,10 @@ # _is_allowed_url_pchars_re = re.compile(r"^[/!$&'()*+,;=:@%a-zA-Z0-9._~-]+$") # We are more lenient for assumed real world compatibility purposes. +# These characters are not allowed within HTTP method names +# to prevent http header injection. +_contains_disallowed_method_pchar_re = re.compile('[\x00-\x1f]') + # We always set the Content-Length header for these methods because some # servers will otherwise respond with a 411 _METHODS_EXPECTING_BODY = {'PATCH', 'POST', 'PUT'} @@ -1102,6 +1106,8 @@ def putrequest(self, method, url, skip_host=False, else: raise CannotSendRequest(self.__state) + self._validate_method(method) + # Save the method for use later in the response phase self._method = method @@ -1192,6 +1198,15 @@ def _encode_request(self, request): # ASCII also helps prevent CVE-2019-9740. return request.encode('ascii') + def _validate_method(self, method): + """Validate a method name for putrequest.""" + # prevent http header injection + match = _contains_disallowed_method_pchar_re.search(method) + if match: + raise ValueError( + f"method can't contain control characters. {method!r} " + f"(found at least {match.group()!r})") + def _validate_path(self, url): """Validate a url for putrequest.""" # Prevent CVE-2019-9740. diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py index 1ac31bf2a8e7f..3431bb80ea6de 100644 --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -368,6 +368,28 @@ def test_headers_debuglevel(self): self.assertEqual(lines[3], "header: Second: val2") +class HttpMethodTests(TestCase): + def test_invalid_method_names(self): + methods = ( + 'GET\r', + 'POST\n', + 'PUT\n\r', + 'POST\nValue', + 'POST\nHOST:abc', + 'GET\nrHost:abc\n', + 'POST\rRemainder:\r', + 'GET\rHOST:\n', + '\nPUT' + ) + + for method in methods: + with self.assertRaisesRegex( + ValueError, "method can't contain control characters"): + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn.request(method=method, url="/") + + class TransferEncodingTest(TestCase): expected_body = b"It's just a flesh wound" diff --git a/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst new file mode 100644 index 0000000000000..990affc3edd9d --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst @@ -0,0 +1,2 @@ +Prevent http header injection by rejecting control characters in +http.client.putrequest(...).

1 0

bpo-41325: Add version note for args and kwargs property in call object (GH-21525)
by Miss Islington (bot) 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/7734738d71c052779d3cb189e5ba0759be… commit: 7734738d71c052779d3cb189e5ba0759beb8d620 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T06:13:21-07:00 summary: bpo-41325: Add version note for args and kwargs property in call object (GH-21525) (cherry picked from commit 9b01c598ca2576a1056816e85dd84bf5f9c74688) Co-authored-by: Jordan Speicher <uSpike(a)users.noreply.github.com> files: M Doc/library/unittest.mock.rst diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index e92f5545d3eb0..34966b38a0dcd 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -647,6 +647,9 @@ the *new_callable* argument to :func:`patch`. arguments and make more complex assertions. See :ref:`calls as tuples <calls-as-tuples>`. + .. versionchanged:: 3.8 + Added ``args`` and ``kwargs`` properties. + .. attribute:: call_args_list

1 0

bpo-41325: Add version note for args and kwargs property in call object (GH-21525)
by Jordan Speicher 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/9b01c598ca2576a1056816e85dd84bf5f9… commit: 9b01c598ca2576a1056816e85dd84bf5f9c74688 branch: master author: Jordan Speicher <uSpike(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T18:35:44+05:30 summary: bpo-41325: Add version note for args and kwargs property in call object (GH-21525) files: M Doc/library/unittest.mock.rst diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 3643d1ad96ed1..c5360f91f518d 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -647,6 +647,9 @@ the *new_callable* argument to :func:`patch`. arguments and make more complex assertions. See :ref:`calls as tuples <calls-as-tuples>`. + .. versionchanged:: 3.8 + Added ``args`` and ``kwargs`` properties. + .. attribute:: call_args_list

1 0

bpo-41304: Update NEWS to include CVE-2020-15801 reference (GH-21521) (GH-21524)
by Miss Islington (bot) 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/eb0d255ffe002412bb937e1bde61225e54… commit: eb0d255ffe002412bb937e1bde61225e5431da5e branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T05:00:57-04:00 summary: bpo-41304: Update NEWS to include CVE-2020-15801 reference (GH-21521) (GH-21524) (cherry picked from commit 164b04c47e61bd35d55e61bc74f9fd646eba81bb) Co-authored-by: Steve Dower <steve.dower(a)python.org> files: M Misc/NEWS.d/next/Security/2020-07-15-20-15-08.bpo-41304.vNEeYA.rst diff --git a/Misc/NEWS.d/next/Security/2020-07-15-20-15-08.bpo-41304.vNEeYA.rst b/Misc/NEWS.d/next/Security/2020-07-15-20-15-08.bpo-41304.vNEeYA.rst index 90423e9a665fc..8cc4bb8d280a7 100644 --- a/Misc/NEWS.d/next/Security/2020-07-15-20-15-08.bpo-41304.vNEeYA.rst +++ b/Misc/NEWS.d/next/Security/2020-07-15-20-15-08.bpo-41304.vNEeYA.rst @@ -1 +1 @@ -Fixes `python3x._pth` being ignored on Windows +Fixes `python3x._pth` being ignored on Windows, caused by the fix for :issue:`29778` (CVE-2020-15801).

1 0

bpo-41262: Convert memoryview to Argument Clinic. (GH-21421)
by Serhiy Storchaka 18 Jul '20

18 Jul '20

https://github.com/python/cpython/commit/80a50368c0e4dc9d56af0ce748dea35c9d… commit: 80a50368c0e4dc9d56af0ce748dea35c9d96d23f branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-07-18T11:12:05+03:00 summary: bpo-41262: Convert memoryview to Argument Clinic. (GH-21421) files: M Objects/clinic/memoryobject.c.h M Objects/memoryobject.c diff --git a/Objects/clinic/memoryobject.c.h b/Objects/clinic/memoryobject.c.h index 8227f0edfbab4..4a682f69d6573 100644 --- a/Objects/clinic/memoryobject.c.h +++ b/Objects/clinic/memoryobject.c.h @@ -2,6 +2,198 @@ preserve [clinic start generated code]*/ +PyDoc_STRVAR(memoryview__doc__, +"memoryview(object)\n" +"--\n" +"\n" +"Create a new memoryview object which references the given object."); + +static PyObject * +memoryview_impl(PyTypeObject *type, PyObject *object); + +static PyObject * +memoryview(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"object", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "memoryview", 0}; + PyObject *argsbuf[1]; + PyObject * const *fastargs; + Py_ssize_t nargs = PyTuple_GET_SIZE(args); + PyObject *object; + + fastargs = _PyArg_UnpackKeywords(_PyTuple_CAST(args)->ob_item, nargs, kwargs, NULL, &_parser, 1, 1, 0, argsbuf); + if (!fastargs) { + goto exit; + } + object = fastargs[0]; + return_value = memoryview_impl(type, object); + +exit: + return return_value; +} + +PyDoc_STRVAR(memoryview_release__doc__, +"release($self, /)\n" +"--\n" +"\n" +"Release the underlying buffer exposed by the memoryview object."); + +#define MEMORYVIEW_RELEASE_METHODDEF \ + {"release", (PyCFunction)memoryview_release, METH_NOARGS, memoryview_release__doc__}, + +static PyObject * +memoryview_release_impl(PyMemoryViewObject *self); + +static PyObject * +memoryview_release(PyMemoryViewObject *self, PyObject *Py_UNUSED(ignored)) +{ + return memoryview_release_impl(self); +} + +PyDoc_STRVAR(memoryview_cast__doc__, +"cast($self, /, format, shape=<unrepresentable>)\n" +"--\n" +"\n" +"Cast a memoryview to a new format or shape."); + +#define MEMORYVIEW_CAST_METHODDEF \ + {"cast", (PyCFunction)(void(*)(void))memoryview_cast, METH_FASTCALL|METH_KEYWORDS, memoryview_cast__doc__}, + +static PyObject * +memoryview_cast_impl(PyMemoryViewObject *self, PyObject *format, + PyObject *shape); + +static PyObject * +memoryview_cast(PyMemoryViewObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"format", "shape", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "cast", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; + PyObject *format; + PyObject *shape = NULL; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 2, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("cast", "argument 'format'", "str", args[0]); + goto exit; + } + if (PyUnicode_READY(args[0]) == -1) { + goto exit; + } + format = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + shape = args[1]; +skip_optional_pos: + return_value = memoryview_cast_impl(self, format, shape); + +exit: + return return_value; +} + +PyDoc_STRVAR(memoryview_toreadonly__doc__, +"toreadonly($self, /)\n" +"--\n" +"\n" +"Return a readonly version of the memoryview."); + +#define MEMORYVIEW_TOREADONLY_METHODDEF \ + {"toreadonly", (PyCFunction)memoryview_toreadonly, METH_NOARGS, memoryview_toreadonly__doc__}, + +static PyObject * +memoryview_toreadonly_impl(PyMemoryViewObject *self); + +static PyObject * +memoryview_toreadonly(PyMemoryViewObject *self, PyObject *Py_UNUSED(ignored)) +{ + return memoryview_toreadonly_impl(self); +} + +PyDoc_STRVAR(memoryview_tolist__doc__, +"tolist($self, /)\n" +"--\n" +"\n" +"Return the data in the buffer as a list of elements."); + +#define MEMORYVIEW_TOLIST_METHODDEF \ + {"tolist", (PyCFunction)memoryview_tolist, METH_NOARGS, memoryview_tolist__doc__}, + +static PyObject * +memoryview_tolist_impl(PyMemoryViewObject *self); + +static PyObject * +memoryview_tolist(PyMemoryViewObject *self, PyObject *Py_UNUSED(ignored)) +{ + return memoryview_tolist_impl(self); +} + +PyDoc_STRVAR(memoryview_tobytes__doc__, +"tobytes($self, /, order=\'C\')\n" +"--\n" +"\n" +"Return the data in the buffer as a byte string.\n" +"\n" +"Order can be {\'C\', \'F\', \'A\'}. When order is \'C\' or \'F\', the data of the\n" +"original array is converted to C or Fortran order. For contiguous views,\n" +"\'A\' returns an exact copy of the physical memory. In particular, in-memory\n" +"Fortran order is preserved. For non-contiguous views, the data is converted\n" +"to C first. order=None is the same as order=\'C\'."); + +#define MEMORYVIEW_TOBYTES_METHODDEF \ + {"tobytes", (PyCFunction)(void(*)(void))memoryview_tobytes, METH_FASTCALL|METH_KEYWORDS, memoryview_tobytes__doc__}, + +static PyObject * +memoryview_tobytes_impl(PyMemoryViewObject *self, const char *order); + +static PyObject * +memoryview_tobytes(PyMemoryViewObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"order", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "tobytes", 0}; + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; + const char *order = NULL; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (!noptargs) { + goto skip_optional_pos; + } + if (args[0] == Py_None) { + order = NULL; + } + else if (PyUnicode_Check(args[0])) { + Py_ssize_t order_length; + order = PyUnicode_AsUTF8AndSize(args[0], &order_length); + if (order == NULL) { + goto exit; + } + if (strlen(order) != (size_t)order_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + } + else { + _PyArg_BadArgument("tobytes", "argument 'order'", "str or None", args[0]); + goto exit; + } +skip_optional_pos: + return_value = memoryview_tobytes_impl(self, order); + +exit: + return return_value; +} + PyDoc_STRVAR(memoryview_hex__doc__, "hex($self, /, sep=<unrepresentable>, bytes_per_sep=1)\n" "--\n" @@ -66,4 +258,4 @@ memoryview_hex(PyMemoryViewObject *self, PyObject *const *args, Py_ssize_t nargs exit: return return_value; } -/*[clinic end generated code: output=91106ef704134b19 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=1b879bb934d18c66 input=a9049054013a1b77]*/ diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c index e3d3bd6a174ff..13d883ae4d35d 100644 --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -238,12 +238,6 @@ PyTypeObject _PyManagedBuffer_Type = { #define REQ_FORMAT(flags) (flags&PyBUF_FORMAT) -PyDoc_STRVAR(memory_doc, -"memoryview(object)\n--\n\ -\n\ -Create a new memoryview object which references the given object."); - - /**************************************************************************/ /* Copy memoryview buffers */ /**************************************************************************/ @@ -961,18 +955,20 @@ PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) } -static PyObject * -memory_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds) -{ - PyObject *obj; - static char *kwlist[] = {"object", NULL}; +/*[clinic input] +@classmethod +memoryview.__new__ - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O:memoryview", kwlist, - &obj)) { - return NULL; - } + object: object + +Create a new memoryview object which references the given object. +[clinic start generated code]*/ - return PyMemoryView_FromObject(obj); +static PyObject * +memoryview_impl(PyTypeObject *type, PyObject *object) +/*[clinic end generated code: output=7de78e184ed66db8 input=f04429eb0bdf8c6e]*/ +{ + return PyMemoryView_FromObject(object); } @@ -1062,8 +1058,15 @@ _memory_release(PyMemoryViewObject *self) return -1; } +/*[clinic input] +memoryview.release + +Release the underlying buffer exposed by the memoryview object. +[clinic start generated code]*/ + static PyObject * -memory_release(PyMemoryViewObject *self, PyObject *noargs) +memoryview_release_impl(PyMemoryViewObject *self) +/*[clinic end generated code: output=d0b7e3ba95b7fcb9 input=bc71d1d51f4a52f0]*/ { if (_memory_release(self) < 0) return NULL; @@ -1108,7 +1111,7 @@ memory_enter(PyObject *self, PyObject *args) static PyObject * memory_exit(PyObject *self, PyObject *args) { - return memory_release((PyMemoryViewObject *)self, NULL); + return memoryview_release_impl((PyMemoryViewObject *)self); } @@ -1352,26 +1355,25 @@ zero_in_shape(PyMemoryViewObject *mv) All casts must result in views that will have the exact byte size of the original input. Otherwise, an error is raised. */ +/*[clinic input] +memoryview.cast + + format: unicode + shape: object = NULL + +Cast a memoryview to a new format or shape. +[clinic start generated code]*/ + static PyObject * -memory_cast(PyMemoryViewObject *self, PyObject *args, PyObject *kwds) +memoryview_cast_impl(PyMemoryViewObject *self, PyObject *format, + PyObject *shape) +/*[clinic end generated code: output=bae520b3a389cbab input=138936cc9041b1a3]*/ { - static char *kwlist[] = {"format", "shape", NULL}; PyMemoryViewObject *mv = NULL; - PyObject *shape = NULL; - PyObject *format; Py_ssize_t ndim = 1; CHECK_RELEASED(self); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O", kwlist, - &format, &shape)) { - return NULL; - } - if (!PyUnicode_Check(format)) { - PyErr_SetString(PyExc_TypeError, - "memoryview: format argument must be a string"); - return NULL; - } if (!MV_C_CONTIGUOUS(self->flags)) { PyErr_SetString(PyExc_TypeError, "memoryview: casts are restricted to C-contiguous views"); @@ -1415,8 +1417,15 @@ memory_cast(PyMemoryViewObject *self, PyObject *args, PyObject *kwds) return NULL; } +/*[clinic input] +memoryview.toreadonly + +Return a readonly version of the memoryview. +[clinic start generated code]*/ + static PyObject * -memory_toreadonly(PyMemoryViewObject *self, PyObject *noargs) +memoryview_toreadonly_impl(PyMemoryViewObject *self) +/*[clinic end generated code: output=2c7e056f04c99e62 input=dc06d20f19ba236f]*/ { CHECK_RELEASED(self); /* Even if self is already readonly, we still need to create a new @@ -2109,13 +2118,20 @@ tolist_rec(const char *ptr, Py_ssize_t ndim, const Py_ssize_t *shape, /* Return a list representation of the memoryview. Currently only buffers with native format strings are supported. */ +/*[clinic input] +memoryview.tolist + +Return the data in the buffer as a list of elements. +[clinic start generated code]*/ + static PyObject * -memory_tolist(PyMemoryViewObject *mv, PyObject *noargs) +memoryview_tolist_impl(PyMemoryViewObject *self) +/*[clinic end generated code: output=a6cda89214fd5a1b input=21e7d0c1860b211a]*/ { - const Py_buffer *view = &(mv->view); + const Py_buffer *view = &self->view; const char *fmt; - CHECK_RELEASED(mv); + CHECK_RELEASED(self); fmt = adjust_fmt(view); if (fmt == NULL) @@ -2135,21 +2151,30 @@ memory_tolist(PyMemoryViewObject *mv, PyObject *noargs) } } +/*[clinic input] +memoryview.tobytes + + order: str(accept={str, NoneType}, c_default="NULL") = 'C' + +Return the data in the buffer as a byte string. + +Order can be {'C', 'F', 'A'}. When order is 'C' or 'F', the data of the +original array is converted to C or Fortran order. For contiguous views, +'A' returns an exact copy of the physical memory. In particular, in-memory +Fortran order is preserved. For non-contiguous views, the data is converted +to C first. order=None is the same as order='C'. +[clinic start generated code]*/ + static PyObject * -memory_tobytes(PyMemoryViewObject *self, PyObject *args, PyObject *kwds) +memoryview_tobytes_impl(PyMemoryViewObject *self, const char *order) +/*[clinic end generated code: output=1288b62560a32a23 input=0efa3ddaeda573a8]*/ { - static char *kwlist[] = {"order", NULL}; Py_buffer *src = VIEW_ADDR(self); - char *order = NULL; char ord = 'C'; PyObject *bytes; CHECK_RELEASED(self); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|z", kwlist, &order)) { - return NULL; - } - if (order) { if (strcmp(order, "F") == 0) { ord = 'F'; @@ -3122,38 +3147,14 @@ static PyGetSetDef memory_getsetlist[] = { {NULL, NULL, NULL, NULL}, }; -PyDoc_STRVAR(memory_release_doc, -"release($self, /)\n--\n\ -\n\ -Release the underlying buffer exposed by the memoryview object."); -PyDoc_STRVAR(memory_tobytes_doc, -"tobytes($self, /, order=None)\n--\n\ -\n\ -Return the data in the buffer as a byte string. Order can be {'C', 'F', 'A'}.\n\ -When order is 'C' or 'F', the data of the original array is converted to C or\n\ -Fortran order. For contiguous views, 'A' returns an exact copy of the physical\n\ -memory. In particular, in-memory Fortran order is preserved. For non-contiguous\n\ -views, the data is converted to C first. order=None is the same as order='C'."); -PyDoc_STRVAR(memory_tolist_doc, -"tolist($self, /)\n--\n\ -\n\ -Return the data in the buffer as a list of elements."); -PyDoc_STRVAR(memory_cast_doc, -"cast($self, /, format, *, shape)\n--\n\ -\n\ -Cast a memoryview to a new format or shape."); -PyDoc_STRVAR(memory_toreadonly_doc, -"toreadonly($self, /)\n--\n\ -\n\ -Return a readonly version of the memoryview."); static PyMethodDef memory_methods[] = { - {"release", (PyCFunction)memory_release, METH_NOARGS, memory_release_doc}, - {"tobytes", (PyCFunction)(void(*)(void))memory_tobytes, METH_VARARGS|METH_KEYWORDS, memory_tobytes_doc}, + MEMORYVIEW_RELEASE_METHODDEF + MEMORYVIEW_TOBYTES_METHODDEF MEMORYVIEW_HEX_METHODDEF - {"tolist", (PyCFunction)memory_tolist, METH_NOARGS, memory_tolist_doc}, - {"cast", (PyCFunction)(void(*)(void))memory_cast, METH_VARARGS|METH_KEYWORDS, memory_cast_doc}, - {"toreadonly", (PyCFunction)memory_toreadonly, METH_NOARGS, memory_toreadonly_doc}, + MEMORYVIEW_TOLIST_METHODDEF + MEMORYVIEW_CAST_METHODDEF + MEMORYVIEW_TOREADONLY_METHODDEF {"__enter__", memory_enter, METH_NOARGS, NULL}, {"__exit__", memory_exit, METH_VARARGS, NULL}, {NULL, NULL} @@ -3181,7 +3182,7 @@ PyTypeObject PyMemoryView_Type = { 0, /* tp_setattro */ &memory_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - memory_doc, /* tp_doc */ + memoryview__doc__, /* tp_doc */ (traverseproc)memory_traverse, /* tp_traverse */ (inquiry)memory_clear, /* tp_clear */ memory_richcompare, /* tp_richcompare */ @@ -3198,5 +3199,5 @@ PyTypeObject PyMemoryView_Type = { 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ - memory_new, /* tp_new */ + memoryview, /* tp_new */ };

1 0