Mailman 3 September 2018 - Python-checkins

bpo-34687: Update asyncio doc for ProactorEventLoop (GH-9623)
by Miss Islington (bot) 28 Sep '18

28 Sep '18

https://github.com/python/cpython/commit/37aae9dcf18753b8ffda99d1a5758a90af… commit: 37aae9dcf18753b8ffda99d1a5758a90af852464 branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> date: 2018-09-28T08:40:08-07:00 summary: bpo-34687: Update asyncio doc for ProactorEventLoop (GH-9623) Since ProactorEventLoop is now the default in 3.8, remove examples using it explicitly on Windows. https://bugs.python.org/issue34687 files: M Doc/library/asyncio-eventloop.rst M Doc/library/asyncio-protocol.rst M Doc/library/asyncio-subprocess.rst diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 50d6ea4b1389..76c8ce9943ef 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -1432,15 +1432,6 @@ on all platforms. Availability: Windows. - An example how to use :class:`ProactorEventLoop` on Windows:: - - import asyncio - import sys - - if sys.platform == 'win32': - loop = asyncio.ProactorEventLoop() - asyncio.set_event_loop(loop) - .. seealso:: `MSDN documentation on I/O Completion Ports diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index bdfdcf7ddb6e..14ec31a6d5d2 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -1030,10 +1030,6 @@ The subprocess is created by th :meth:`loop.subprocess_exec` method:: data = bytes(protocol.output) return data.decode('ascii').rstrip() - if sys.platform == "win32": - asyncio.set_event_loop_policy( - asyncio.WindowsProactorEventLoopPolicy()) - date = asyncio.run(get_date()) print(f"Current date: {date}") diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index 0bcf66175ce3..a2cf51734848 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -344,10 +344,6 @@ function:: await proc.wait() return line - if sys.platform == "win32": - asyncio.set_event_loop_policy( - asyncio.WindowsProactorEventLoopPolicy()) - date = asyncio.run(get_date()) print(f"Current date: {date}")

1 0

bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613) (GH-9619)
by Victor Stinner 28 Sep '18

28 Sep '18

https://github.com/python/cpython/commit/007fda436f707ac95c2fa8f8886efd9e09… commit: 007fda436f707ac95c2fa8f8886efd9e09d5b630 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Victor Stinner <vstinner(a)redhat.com> date: 2018-09-28T06:07:56-07:00 summary: bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613) (GH-9619) After some failures in AMD64 FreeBSD CURRENT Debug 3.x buildbots regarding tests in test_multiprocessing_spawn and after examining similar failures in test_socket, some errors in the calculation of ancillary data buffers were found in multiprocessing.reduction. CMSG_LEN() can often be used as the buffer size for recvmsg() to receive a single item of ancillary data, but RFC 3542 requires portable applications to use CMSG_SPACE() and thus include space for padding, even when the item will be the last in the buffer. The failures we experience are due to the usage of CMSG_LEN() instead of CMSG_SPACE(). (cherry picked from commit 077061a7b24917aaf31057885c69919c5a553c88) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: M Lib/multiprocessing/reduction.py diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py index deca19ccad7b..0f5432593a58 100644 --- a/Lib/multiprocessing/reduction.py +++ b/Lib/multiprocessing/reduction.py @@ -150,7 +150,7 @@ def recvfds(sock, size): '''Receive an array of fds over an AF_UNIX socket.''' a = array.array('i') bytes_size = a.itemsize * size - msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) + msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_SPACE(bytes_size)) if not msg and not ancdata: raise EOFError try:

1 0

bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613)
by Miss Islington (bot) 28 Sep '18

28 Sep '18

https://github.com/python/cpython/commit/5d33ee1595767d68800fda13bc68b7b01d… commit: 5d33ee1595767d68800fda13bc68b7b01dba5117 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-09-28T04:31:51-07:00 summary: bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613) After some failures in AMD64 FreeBSD CURRENT Debug 3.x buildbots regarding tests in test_multiprocessing_spawn and after examining similar failures in test_socket, some errors in the calculation of ancillary data buffers were found in multiprocessing.reduction. CMSG_LEN() can often be used as the buffer size for recvmsg() to receive a single item of ancillary data, but RFC 3542 requires portable applications to use CMSG_SPACE() and thus include space for padding, even when the item will be the last in the buffer. The failures we experience are due to the usage of CMSG_LEN() instead of CMSG_SPACE(). (cherry picked from commit 077061a7b24917aaf31057885c69919c5a553c88) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: M Lib/multiprocessing/reduction.py diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py index c043c9a0dc16..f1b9d3ac99ce 100644 --- a/Lib/multiprocessing/reduction.py +++ b/Lib/multiprocessing/reduction.py @@ -150,7 +150,7 @@ def recvfds(sock, size): '''Receive an array of fds over an AF_UNIX socket.''' a = array.array('i') bytes_size = a.itemsize * size - msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) + msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_SPACE(bytes_size)) if not msg and not ancdata: raise EOFError try:

1 0

bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613)
by Victor Stinner 28 Sep '18

28 Sep '18

https://github.com/python/cpython/commit/077061a7b24917aaf31057885c69919c5a… commit: 077061a7b24917aaf31057885c69919c5a553c88 branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: Victor Stinner <vstinner(a)redhat.com> date: 2018-09-28T02:51:05-07:00 summary: bpo-34521: Fix tests in test_multiprocessing_spawn to use correctly CMSG_SPACE (GH-9613) After some failures in AMD64 FreeBSD CURRENT Debug 3.x buildbots regarding tests in test_multiprocessing_spawn and after examining similar failures in test_socket, some errors in the calculation of ancillary data buffers were found in multiprocessing.reduction. CMSG_LEN() can often be used as the buffer size for recvmsg() to receive a single item of ancillary data, but RFC 3542 requires portable applications to use CMSG_SPACE() and thus include space for padding, even when the item will be the last in the buffer. The failures we experience are due to the usage of CMSG_LEN() instead of CMSG_SPACE(). files: M Lib/multiprocessing/reduction.py diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py index 473fd59df61b..5593f0682f7f 100644 --- a/Lib/multiprocessing/reduction.py +++ b/Lib/multiprocessing/reduction.py @@ -154,7 +154,7 @@ def recvfds(sock, size): '''Receive an array of fds over an AF_UNIX socket.''' a = array.array('i') bytes_size = a.itemsize * size - msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size)) + msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_SPACE(bytes_size)) if not msg and not ancdata: raise EOFError try:

1 0

Daily reference leaks (4243df51fe43): sum=-2
by solipsis＠pitrou.net 28 Sep '18

28 Sep '18

results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [0, -7, 1] memory blocks, sum=-6 test_functools leaked [0, 3, 1] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogKJ5bcf', '--timeout', '7200']

1 0

bpo-34736: improve error message for invalid length b64decode inputs (GH-9563)
by Miss Islington (bot) 27 Sep '18

27 Sep '18

https://github.com/python/cpython/commit/7e35081bc828291da5793db49ab45dee4f… commit: 7e35081bc828291da5793db49ab45dee4fda5043 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-09-27T23:12:54-07:00 summary: bpo-34736: improve error message for invalid length b64decode inputs (GH-9563) Improvements: 1. Include the number of valid data characters in the error message. 2. Mention "number of data characters" rather than "length". https://bugs.python.org/issue34736 (cherry picked from commit 1fba2ffc37da52c08db51fe4360459990b0311c9) Co-authored-by: Tal Einat <taleinat+github(a)gmail.com> files: M Lib/test/test_binascii.py M Modules/binascii.c diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py index 7418a9ce9848..8e37d55a64f9 100644 --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -3,6 +3,7 @@ import unittest import binascii import array +import re # Note: "*_hex" functions are aliases for "(un)hexlify" b2a_functions = ['b2a_base64', 'b2a_hex', 'b2a_hqx', 'b2a_qp', 'b2a_uu', @@ -127,7 +128,10 @@ def assertIncorrectPadding(data): # Test base64 with invalid number of valid characters (1 mod 4) def assertInvalidLength(data): - with self.assertRaisesRegex(binascii.Error, r'(?i)invalid.+length'): + n_data_chars = len(re.sub(br'[^A-Za-z0-9/+]', br'', data)) + expected_errmsg_re = \ + r'(?i)Invalid.+number of data characters.+' + str(n_data_chars) + with self.assertRaisesRegex(binascii.Error, expected_errmsg_re): binascii.a2b_base64(self.type2test(data)) assertInvalidLength(b'a') diff --git a/Modules/binascii.c b/Modules/binascii.c index 2df80affb695..d0d3f7d34b8f 100644 --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -438,6 +438,7 @@ binascii_a2b_base64_impl(PyObject *module, Py_buffer *data) { const unsigned char *ascii_data; unsigned char *bin_data; + unsigned char *bin_data_start; int leftbits = 0; unsigned char this_ch; unsigned int leftchar = 0; @@ -461,6 +462,7 @@ binascii_a2b_base64_impl(PyObject *module, Py_buffer *data) bin_data = _PyBytesWriter_Alloc(&writer, bin_len); if (bin_data == NULL) return NULL; + bin_data_start = bin_data; for( ; ascii_len > 0; ascii_len--, ascii_data++) { this_ch = *ascii_data; @@ -516,9 +518,11 @@ binascii_a2b_base64_impl(PyObject *module, Py_buffer *data) ** This is an invalid length, as there is no possible input that ** could encoded into such a base64 string. */ - PyErr_SetString(Error, - "Invalid base64-encoded string: " - "length cannot be 1 more than a multiple of 4"); + PyErr_Format(Error, + "Invalid base64-encoded string: " + "number of data characters (%d) cannot be 1 more " + "than a multiple of 4", + (bin_data - bin_data_start) / 3 * 4 + 1); } else { PyErr_SetString(Error, "Incorrect padding"); }

1 0

bpo-34736: improve error message for invalid length b64decode inputs (GH-9563)
by Miss Islington (bot) 27 Sep '18

27 Sep '18

https://github.com/python/cpython/commit/1fba2ffc37da52c08db51fe4360459990b… commit: 1fba2ffc37da52c08db51fe4360459990b0311c9 branch: master author: Tal Einat <taleinat+github(a)gmail.com> committer: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> date: 2018-09-27T22:57:22-07:00 summary: bpo-34736: improve error message for invalid length b64decode inputs (GH-9563) Improvements: 1. Include the number of valid data characters in the error message. 2. Mention "number of data characters" rather than "length". https://bugs.python.org/issue34736 files: M Lib/test/test_binascii.py M Modules/binascii.c diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py index c5fcc1ac1647..572e50c3e25e 100644 --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -3,6 +3,7 @@ import unittest import binascii import array +import re # Note: "*_hex" functions are aliases for "(un)hexlify" b2a_functions = ['b2a_base64', 'b2a_hex', 'b2a_hqx', 'b2a_qp', 'b2a_uu', @@ -127,7 +128,10 @@ def assertIncorrectPadding(data): # Test base64 with invalid number of valid characters (1 mod 4) def assertInvalidLength(data): - with self.assertRaisesRegex(binascii.Error, r'(?i)invalid.+length'): + n_data_chars = len(re.sub(br'[^A-Za-z0-9/+]', br'', data)) + expected_errmsg_re = \ + r'(?i)Invalid.+number of data characters.+' + str(n_data_chars) + with self.assertRaisesRegex(binascii.Error, expected_errmsg_re): binascii.a2b_base64(self.type2test(data)) assertInvalidLength(b'a') diff --git a/Modules/binascii.c b/Modules/binascii.c index 5667018d6e1b..19589f906f72 100644 --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -438,6 +438,7 @@ binascii_a2b_base64_impl(PyObject *module, Py_buffer *data) { const unsigned char *ascii_data; unsigned char *bin_data; + unsigned char *bin_data_start; int leftbits = 0; unsigned char this_ch; unsigned int leftchar = 0; @@ -461,6 +462,7 @@ binascii_a2b_base64_impl(PyObject *module, Py_buffer *data) bin_data = _PyBytesWriter_Alloc(&writer, bin_len); if (bin_data == NULL) return NULL; + bin_data_start = bin_data; for( ; ascii_len > 0; ascii_len--, ascii_data++) { this_ch = *ascii_data; @@ -516,9 +518,11 @@ binascii_a2b_base64_impl(PyObject *module, Py_buffer *data) ** This is an invalid length, as there is no possible input that ** could encoded into such a base64 string. */ - PyErr_SetString(Error, - "Invalid base64-encoded string: " - "length cannot be 1 more than a multiple of 4"); + PyErr_Format(Error, + "Invalid base64-encoded string: " + "number of data characters (%d) cannot be 1 more " + "than a multiple of 4", + (bin_data - bin_data_start) / 3 * 4 + 1); } else { PyErr_SetString(Error, "Incorrect padding"); }

1 0

bpo-34248: Add filename to error raised in {gnu, ndbm}.open() (GH-8590)
by Berker Peksag 27 Sep '18

27 Sep '18

https://github.com/python/cpython/commit/9df346bf98069a87de14a3c2f69009d800… commit: 9df346bf98069a87de14a3c2f69009d800994c63 branch: master author: Zsolt Cserna <cserna.zsolt(a)gmail.com> committer: Berker Peksag <berker.peksag(a)gmail.com> date: 2018-09-27T22:54:34+03:00 summary: bpo-34248: Add filename to error raised in {gnu,ndbm}.open() (GH-8590) Report the filename to the exception when raising {gdbm,dbm.ndbm}.error in dbm.gnu.open() and dbm.ndbm.open() functions, so it gets printed when the exception is raised, and can also be obtained by the filename attribute of the exception object. files: A Misc/NEWS.d/next/Library/2018-07-31-23-00-09.bpo-34248.5U6wwc.rst M Lib/test/test_dbm_gnu.py M Lib/test/test_dbm_ndbm.py M Modules/_dbmmodule.c M Modules/_gdbmmodule.c diff --git a/Lib/test/test_dbm_gnu.py b/Lib/test/test_dbm_gnu.py index c96eff5a319e..16b7fe6fac17 100644 --- a/Lib/test/test_dbm_gnu.py +++ b/Lib/test/test_dbm_gnu.py @@ -144,6 +144,13 @@ def test_nonascii_filename(self): self.assertTrue(b'key' in db) self.assertEqual(db[b'key'], b'value') + def test_nonexisting_file(self): + nonexisting_file = 'nonexisting-file' + with self.assertRaises(gdbm.error) as cm: + gdbm.open(nonexisting_file) + self.assertIn(nonexisting_file, str(cm.exception)) + self.assertEqual(cm.exception.filename, nonexisting_file) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_dbm_ndbm.py b/Lib/test/test_dbm_ndbm.py index 49b88f5cccef..bd411da652e6 100644 --- a/Lib/test/test_dbm_ndbm.py +++ b/Lib/test/test_dbm_ndbm.py @@ -105,6 +105,12 @@ def test_nonascii_filename(self): self.assertTrue(b'key' in db) self.assertEqual(db[b'key'], b'value') + def test_nonexisting_file(self): + nonexisting_file = 'nonexisting-file' + with self.assertRaises(dbm.ndbm.error) as cm: + dbm.ndbm.open(nonexisting_file) + self.assertIn(nonexisting_file, str(cm.exception)) + self.assertEqual(cm.exception.filename, nonexisting_file) if __name__ == '__main__': diff --git a/Misc/NEWS.d/next/Library/2018-07-31-23-00-09.bpo-34248.5U6wwc.rst b/Misc/NEWS.d/next/Library/2018-07-31-23-00-09.bpo-34248.5U6wwc.rst new file mode 100644 index 000000000000..55215f60550b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-07-31-23-00-09.bpo-34248.5U6wwc.rst @@ -0,0 +1,3 @@ +Report filename in the exception raised when the database file cannot be opened +by :func:`dbm.gnu.open` and :func:`dbm.ndbm.open` due to OS-related error. +Patch by Zsolt Cserna. diff --git a/Modules/_dbmmodule.c b/Modules/_dbmmodule.c index 65761d83d380..081184a0b322 100644 --- a/Modules/_dbmmodule.c +++ b/Modules/_dbmmodule.c @@ -62,7 +62,7 @@ newdbmobject(const char *file, int flags, int mode) dp->di_size = -1; /* See issue #19296 */ if ( (dp->di_dbm = dbm_open((char *)file, flags, mode)) == 0 ) { - PyErr_SetFromErrno(DbmError); + PyErr_SetFromErrnoWithFilename(DbmError, file); Py_DECREF(dp); return NULL; } diff --git a/Modules/_gdbmmodule.c b/Modules/_gdbmmodule.c index 10560040e446..ceb744b99ba8 100644 --- a/Modules/_gdbmmodule.c +++ b/Modules/_gdbmmodule.c @@ -75,7 +75,7 @@ newdbmobject(const char *file, int flags, int mode) errno = 0; if ((dp->di_dbm = gdbm_open((char *)file, 0, flags, mode, NULL)) == 0) { if (errno != 0) - PyErr_SetFromErrno(DbmError); + PyErr_SetFromErrnoWithFilename(DbmError, file); else PyErr_SetString(DbmError, gdbm_strerror(gdbm_errno)); Py_DECREF(dp);

1 0

bpo-34802: Fix asyncio.iscoroutine() docs (GH-9611)
by Miss Islington (bot) 27 Sep '18

27 Sep '18

https://github.com/python/cpython/commit/85ccedc5b57ddda198e7176ba787e38964… commit: 85ccedc5b57ddda198e7176ba787e3896435c504 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-09-27T12:53:18-07:00 summary: bpo-34802: Fix asyncio.iscoroutine() docs (GH-9611) (cherry picked from commit 59ee5b12938efbf534f2a19300a847bf6b23a77d) Co-authored-by: Yury Selivanov <yury(a)magic.io> files: M Doc/library/asyncio-task.rst diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index 5cbdfeff32f5..198bd7d07a01 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -893,8 +893,7 @@ enforced. Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`. This method is different from :func:`inspect.iscoroutine` because - it returns ``True`` for generator-based coroutines decorated with - :func:`@coroutine <coroutine>`. + it returns ``True`` for generator-based coroutines. .. function:: iscoroutinefunction(func)

1 0

bpo-34802: Fix asyncio.iscoroutine() docs (GH-9611)
by Yury Selivanov 27 Sep '18

27 Sep '18

https://github.com/python/cpython/commit/59ee5b12938efbf534f2a19300a847bf6b… commit: 59ee5b12938efbf534f2a19300a847bf6b23a77d branch: master author: Yury Selivanov <yury(a)magic.io> committer: GitHub <noreply(a)github.com> date: 2018-09-27T15:48:30-04:00 summary: bpo-34802: Fix asyncio.iscoroutine() docs (GH-9611) files: M Doc/library/asyncio-task.rst diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index 1a504363946e..d7102b807b60 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -924,8 +924,7 @@ enforced. Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`. This method is different from :func:`inspect.iscoroutine` because - it returns ``True`` for generator-based coroutines decorated with - :func:`@coroutine <coroutine>`. + it returns ``True`` for generator-based coroutines. .. function:: iscoroutinefunction(func)

1 0