Mailman 3 October 2023 - Python-checkins

[3.11] gh-101100: Fix sphinx warnings in `library/site.rst` (GH-110144) (#110185)
by hugovk Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/00da3ccd0a7b4f0cfb04e94bcd766326d1… commit: 00da3ccd0a7b4f0cfb04e94bcd766326d12a5af1 branch: 3.11 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: hugovk <hugovk(a)users.noreply.github.com> date: 2023-10-01T11:47:06-06:00 summary: [3.11] gh-101100: Fix sphinx warnings in `library/site.rst` (GH-110144) (#110185) Co-authored-by: Nikita Sobolev <mail(a)sobolevn.me> Co-authored-by: Alex Waygood <Alex.Waygood(a)Gmail.com> files: M Doc/library/exceptions.rst M Doc/library/site.rst M Doc/tools/.nitignore diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index dcfbc486eeb35..d210f82f3926f 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -220,10 +220,16 @@ The following exceptions are the exceptions that are usually raised. load a module. Also raised when the "from list" in ``from ... import`` has a name that cannot be found. - The :attr:`name` and :attr:`path` attributes can be set using keyword-only - arguments to the constructor. When set they represent the name of the module - that was attempted to be imported and the path to any file which triggered - the exception, respectively. + The optional *name* and *path* keyword-only arguments + set the corresponding attributes: + + .. attribute:: name + + The name of the module that was attempted to be imported. + + .. attribute:: path + + The path to any file which triggered the exception. .. versionchanged:: 3.3 Added the :attr:`name` and :attr:`path` attributes. diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 5b70dbced13e0..c025af608e318 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -19,7 +19,7 @@ Importing this module will append site-specific paths to the module search path and add a few builtins, unless :option:`-S` was used. In that case, this module can be safely imported with no automatic modifications to the module search path or additions to the builtins. To explicitly trigger the usual site-specific -additions, call the :func:`site.main` function. +additions, call the :func:`main` function. .. versionchanged:: 3.3 Importing the module used to trigger paths manipulation even when using @@ -109,32 +109,40 @@ directory precedes the :file:`foo` directory because :file:`bar.pth` comes alphabetically before :file:`foo.pth`; and :file:`spam` is omitted because it is not mentioned in either path configuration file. -.. index:: pair: module; sitecustomize +:mod:`sitecustomize` +-------------------- + +.. module:: sitecustomize After these path manipulations, an attempt is made to import a module named :mod:`sitecustomize`, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in the site-packages directory. If this import fails with an :exc:`ImportError` or its subclass -exception, and the exception's :attr:`name` attribute equals to ``'sitecustomize'``, +exception, and the exception's :attr:`~ImportError.name` +attribute equals to ``'sitecustomize'``, it is silently ignored. If Python is started without output streams available, as with :file:`pythonw.exe` on Windows (which is used by default to start IDLE), attempted output from :mod:`sitecustomize` is ignored. Any other exception causes a silent and perhaps mysterious failure of the process. -.. index:: pair: module; usercustomize +:mod:`usercustomize` +-------------------- + +.. module:: usercustomize After this, an attempt is made to import a module named :mod:`usercustomize`, which can perform arbitrary user-specific customizations, if -:data:`ENABLE_USER_SITE` is true. This file is intended to be created in the +:data:`~site.ENABLE_USER_SITE` is true. This file is intended to be created in the user site-packages directory (see below), which is part of ``sys.path`` unless disabled by :option:`-s`. If this import fails with an :exc:`ImportError` or -its subclass exception, and the exception's :attr:`name` attribute equals to -``'usercustomize'``, it is silently ignored. +its subclass exception, and the exception's :attr:`~ImportError.name` +attribute equals to ``'usercustomize'``, it is silently ignored. Note that for some non-Unix systems, ``sys.prefix`` and ``sys.exec_prefix`` are empty, and the path manipulations are skipped; however the import of :mod:`sitecustomize` and :mod:`usercustomize` is still attempted. +.. currentmodule:: site .. _rlcompleter-config: diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index dace135cb560f..8e1c28a4c7b42 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -110,7 +110,6 @@ Doc/library/select.rst Doc/library/selectors.rst Doc/library/shelve.rst Doc/library/signal.rst -Doc/library/site.rst Doc/library/smtplib.rst Doc/library/socket.rst Doc/library/socketserver.rst @@ -118,7 +117,6 @@ Doc/library/ssl.rst Doc/library/stdtypes.rst Doc/library/string.rst Doc/library/subprocess.rst -Doc/library/sys_path_init.rst Doc/library/syslog.rst Doc/library/tarfile.rst Doc/library/tempfile.rst

1 0

gh-101100: Fix sphinx warnings in `library/site.rst` (#110144)
by hugovk Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/31097df611bb5c8084190202e095ae47e8… commit: 31097df611bb5c8084190202e095ae47e8b81c0f branch: main author: Nikita Sobolev <mail(a)sobolevn.me> committer: hugovk <hugovk(a)users.noreply.github.com> date: 2023-10-01T20:18:19+03:00 summary: gh-101100: Fix sphinx warnings in `library/site.rst` (#110144) Co-authored-by: Alex Waygood <Alex.Waygood(a)Gmail.com> files: M Doc/library/exceptions.rst M Doc/library/site.rst M Doc/tools/.nitignore diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index fae0cf621323c..cd85df8723a76 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -220,10 +220,16 @@ The following exceptions are the exceptions that are usually raised. load a module. Also raised when the "from list" in ``from ... import`` has a name that cannot be found. - The :attr:`name` and :attr:`path` attributes can be set using keyword-only - arguments to the constructor. When set they represent the name of the module - that was attempted to be imported and the path to any file which triggered - the exception, respectively. + The optional *name* and *path* keyword-only arguments + set the corresponding attributes: + + .. attribute:: name + + The name of the module that was attempted to be imported. + + .. attribute:: path + + The path to any file which triggered the exception. .. versionchanged:: 3.3 Added the :attr:`name` and :attr:`path` attributes. diff --git a/Doc/library/site.rst b/Doc/library/site.rst index ea3b2e996574e..ebd78917a0137 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -19,7 +19,7 @@ Importing this module will append site-specific paths to the module search path and add a few builtins, unless :option:`-S` was used. In that case, this module can be safely imported with no automatic modifications to the module search path or additions to the builtins. To explicitly trigger the usual site-specific -additions, call the :func:`site.main` function. +additions, call the :func:`main` function. .. versionchanged:: 3.3 Importing the module used to trigger paths manipulation even when using @@ -109,32 +109,40 @@ directory precedes the :file:`foo` directory because :file:`bar.pth` comes alphabetically before :file:`foo.pth`; and :file:`spam` is omitted because it is not mentioned in either path configuration file. -.. index:: pair: module; sitecustomize +:mod:`sitecustomize` +-------------------- + +.. module:: sitecustomize After these path manipulations, an attempt is made to import a module named :mod:`sitecustomize`, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in the site-packages directory. If this import fails with an :exc:`ImportError` or its subclass -exception, and the exception's :attr:`name` attribute equals to ``'sitecustomize'``, +exception, and the exception's :attr:`~ImportError.name` +attribute equals to ``'sitecustomize'``, it is silently ignored. If Python is started without output streams available, as with :file:`pythonw.exe` on Windows (which is used by default to start IDLE), attempted output from :mod:`sitecustomize` is ignored. Any other exception causes a silent and perhaps mysterious failure of the process. -.. index:: pair: module; usercustomize +:mod:`usercustomize` +-------------------- + +.. module:: usercustomize After this, an attempt is made to import a module named :mod:`usercustomize`, which can perform arbitrary user-specific customizations, if -:data:`ENABLE_USER_SITE` is true. This file is intended to be created in the +:data:`~site.ENABLE_USER_SITE` is true. This file is intended to be created in the user site-packages directory (see below), which is part of ``sys.path`` unless disabled by :option:`-s`. If this import fails with an :exc:`ImportError` or -its subclass exception, and the exception's :attr:`name` attribute equals to -``'usercustomize'``, it is silently ignored. +its subclass exception, and the exception's :attr:`~ImportError.name` +attribute equals to ``'usercustomize'``, it is silently ignored. Note that for some non-Unix systems, ``sys.prefix`` and ``sys.exec_prefix`` are empty, and the path manipulations are skipped; however the import of :mod:`sitecustomize` and :mod:`usercustomize` is still attempted. +.. currentmodule:: site .. _rlcompleter-config: diff --git a/Doc/tools/.nitignore b/Doc/tools/.nitignore index f217da9052ca7..fbc9fc33ecea1 100644 --- a/Doc/tools/.nitignore +++ b/Doc/tools/.nitignore @@ -106,7 +106,6 @@ Doc/library/select.rst Doc/library/selectors.rst Doc/library/shelve.rst Doc/library/signal.rst -Doc/library/site.rst Doc/library/smtplib.rst Doc/library/socket.rst Doc/library/socketserver.rst @@ -114,7 +113,6 @@ Doc/library/ssl.rst Doc/library/stdtypes.rst Doc/library/string.rst Doc/library/subprocess.rst -Doc/library/sys_path_init.rst Doc/library/syslog.rst Doc/library/tarfile.rst Doc/library/tempfile.rst

1 0

build(deps): bump hypothesis from 6.84.0 to 6.87.1 in /Tools (#110174)
by AlexWaygood Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/a431a0f988f26dae387f2a107c7e695f5d… commit: a431a0f988f26dae387f2a107c7e695f5dfc0096 branch: main author: dependabot[bot] <49699333+dependabot[bot](a)users.noreply.github.com> committer: AlexWaygood <Alex.Waygood(a)Gmail.com> date: 2023-10-01T17:25:29+01:00 summary: build(deps): bump hypothesis from 6.84.0 to 6.87.1 in /Tools (#110174) Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.84.0 to 6.87.1. - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-p… --- updated-dependencies: - dependency-name: hypothesis dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support(a)github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot](a)users.noreply.github.com> files: M Tools/requirements-hypothesis.txt diff --git a/Tools/requirements-hypothesis.txt b/Tools/requirements-hypothesis.txt index 9db2b74c87cfb..b95300a07dd2b 100644 --- a/Tools/requirements-hypothesis.txt +++ b/Tools/requirements-hypothesis.txt @@ -1,4 +1,4 @@ # Requirements file for hypothesis that # we use to run our property-based tests in CI. -hypothesis==6.84.0 +hypothesis==6.87.1

1 0

gh-110180: Remove unused `_PickleUsingNameMixin` class from `typing` (#110181)
by AlexWaygood Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/d642c5bbf58b42c90053dc553885445d53… commit: d642c5bbf58b42c90053dc553885445d53f247fe branch: main author: Alex Waygood <Alex.Waygood(a)Gmail.com> committer: AlexWaygood <Alex.Waygood(a)Gmail.com> date: 2023-10-01T16:28:02+01:00 summary: gh-110180: Remove unused `_PickleUsingNameMixin` class from `typing` (#110181) files: M Lib/typing.py diff --git a/Lib/typing.py b/Lib/typing.py index 639be75747dae..d1f371377b88f 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -937,13 +937,6 @@ def _is_typevar_like(x: Any) -> bool: return isinstance(x, (TypeVar, ParamSpec)) or _is_unpacked_typevartuple(x) -class _PickleUsingNameMixin: - """Mixin enabling pickling based on self.__name__.""" - - def __reduce__(self): - return self.__name__ - - def _typevar_subst(self, arg): msg = "Parameters to generic types must be types." arg = _type_check(arg, msg, is_argument=True)

1 0

GH-107465: Add `pathlib.Path.from_uri()` classmethod. (#107640)
by barneygale Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/15de493395c3251b8b82063bbe22a37979… commit: 15de493395c3251b8b82063bbe22a379792b9404 branch: main author: Barney Gale <barney.gale(a)gmail.com> committer: barneygale <barney.gale(a)gmail.com> date: 2023-10-01T16:14:02+01:00 summary: GH-107465: Add `pathlib.Path.from_uri()` classmethod. (#107640) This method supports file URIs (including variants) as described in RFC 8089, such as URIs generated by `pathlib.Path.as_uri()` and `urllib.request.pathname2url()`. The method is added to `Path` rather than `PurePath` because it uses `os.fsdecode()`, and so its results vary from system to system. I intend to deprecate `PurePath.as_uri()` and move it to `Path` for the same reason. Co-authored-by: Adam Turner <9087854+AA-Turner(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2023-08-04-19-00-53.gh-issue-107465.Vc1Il3.rst M Doc/library/pathlib.rst M Doc/whatsnew/3.13.rst M Lib/pathlib.py M Lib/test/test_pathlib.py diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 48d6176d26bb8..8ee89a003a339 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -850,6 +850,42 @@ call fails (for example because the path doesn't exist). .. versionadded:: 3.5 +.. classmethod:: Path.from_uri(uri) + + Return a new path object from parsing a 'file' URI conforming to + :rfc:`8089`. For example:: + + >>> p = Path.from_uri('file:///etc/hosts') + PosixPath('/etc/hosts') + + On Windows, DOS device and UNC paths may be parsed from URIs:: + + >>> p = Path.from_uri('file:///c:/windows') + WindowsPath('c:/windows') + >>> p = Path.from_uri('file://server/share') + WindowsPath('//server/share') + + Several variant forms are supported:: + + >>> p = Path.from_uri('file:////server/share') + WindowsPath('//server/share') + >>> p = Path.from_uri('file://///server/share') + WindowsPath('//server/share') + >>> p = Path.from_uri('file:c:/windows') + WindowsPath('c:/windows') + >>> p = Path.from_uri('file:/c|/windows') + WindowsPath('c:/windows') + + :exc:`ValueError` is raised if the URI does not start with ``file:``, or + the parsed path isn't absolute. + + :func:`os.fsdecode` is used to decode percent-escaped byte sequences, and + so file URIs are not portable across machines with different + :ref:`filesystem encodings <filesystem-encoding>`. + + .. versionadded:: 3.13 + + .. method:: Path.stat(*, follow_symlinks=True) Return a :class:`os.stat_result` object containing information about this path, like :func:`os.stat`. diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst index a789084a79c39..1de5479a92437 100644 --- a/Doc/whatsnew/3.13.rst +++ b/Doc/whatsnew/3.13.rst @@ -184,6 +184,10 @@ pathlib :exc:`NotImplementedError` when a path operation isn't supported. (Contributed by Barney Gale in :gh:`89812`.) +* Add :meth:`pathlib.Path.from_uri`, a new constructor to create a :class:`pathlib.Path` + object from a 'file' URI (``file:/``). + (Contributed by Barney Gale in :gh:`107465`.) + * Add support for recursive wildcards in :meth:`pathlib.PurePath.match`. (Contributed by Barney Gale in :gh:`73435`.) diff --git a/Lib/pathlib.py b/Lib/pathlib.py index e6be9061013a8..9e6d0754eccf3 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -18,7 +18,6 @@ from _collections_abc import Sequence from errno import ENOENT, ENOTDIR, EBADF, ELOOP, EINVAL from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO -from urllib.parse import quote_from_bytes as urlquote_from_bytes try: import pwd @@ -452,7 +451,8 @@ def as_uri(self): # It's a posix path => 'file:///etc/hosts' prefix = 'file://' path = str(self) - return prefix + urlquote_from_bytes(os.fsencode(path)) + from urllib.parse import quote_from_bytes + return prefix + quote_from_bytes(os.fsencode(path)) @property def _str_normcase(self): @@ -814,9 +814,10 @@ class _PathBase(PurePath): __bytes__ = None __fspath__ = None # virtual paths have no local file system representation - def _unsupported(self, method_name): - msg = f"{type(self).__name__}.{method_name}() is unsupported" - if isinstance(self, Path): + @classmethod + def _unsupported(cls, method_name): + msg = f"{cls.__name__}.{method_name}() is unsupported" + if issubclass(cls, Path): msg += " on this system" raise UnsupportedOperation(msg) @@ -1418,6 +1419,11 @@ def group(self): """ self._unsupported("group") + @classmethod + def from_uri(cls, uri): + """Return a new path from the given 'file' URI.""" + cls._unsupported("from_uri") + def as_uri(self): """Return the path as a URI.""" self._unsupported("as_uri") @@ -1661,6 +1667,30 @@ def expanduser(self): return self + @classmethod + def from_uri(cls, uri): + """Return a new path from the given 'file' URI.""" + if not uri.startswith('file:'): + raise ValueError(f"URI does not start with 'file:': {uri!r}") + path = uri[5:] + if path[:3] == '///': + # Remove empty authority + path = path[2:] + elif path[:12] == '//localhost/': + # Remove 'localhost' authority + path = path[11:] + if path[:3] == '///' or (path[:1] == '/' and path[2:3] in ':|'): + # Remove slash before DOS device/UNC path + path = path[1:] + if path[1:2] == '|': + # Replace bar with colon in DOS drive + path = path[:1] + ':' + path[2:] + from urllib.parse import unquote_to_bytes + path = cls(os.fsdecode(unquote_to_bytes(path))) + if not path.is_absolute(): + raise ValueError(f"URI is not absolute: {uri!r}") + return path + class PosixPath(Path, PurePosixPath): """Path subclass for non-Windows systems. diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 319148e9065a6..76918addf8b61 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -11,6 +11,7 @@ import tempfile import unittest from unittest import mock +from urllib.request import pathname2url from test.support import import_helper from test.support import set_recursion_limit @@ -3602,6 +3603,24 @@ def test_handling_bad_descriptor(self): self.fail("Bad file descriptor not handled.") raise + def test_from_uri(self): + P = self.cls + self.assertEqual(P.from_uri('file:/foo/bar'), P('/foo/bar')) + self.assertEqual(P.from_uri('file://foo/bar'), P('//foo/bar')) + self.assertEqual(P.from_uri('file:///foo/bar'), P('/foo/bar')) + self.assertEqual(P.from_uri('file:////foo/bar'), P('//foo/bar')) + self.assertEqual(P.from_uri('file://localhost/foo/bar'), P('/foo/bar')) + self.assertRaises(ValueError, P.from_uri, 'foo/bar') + self.assertRaises(ValueError, P.from_uri, '/foo/bar') + self.assertRaises(ValueError, P.from_uri, '//foo/bar') + self.assertRaises(ValueError, P.from_uri, 'file:foo/bar') + self.assertRaises(ValueError, P.from_uri, 'http://foo/bar') + + def test_from_uri_pathname2url(self): + P = self.cls + self.assertEqual(P.from_uri('file:' + pathname2url('/foo/bar')), P('/foo/bar')) + self.assertEqual(P.from_uri('file:' + pathname2url('//foo/bar')), P('//foo/bar')) + @only_nt class WindowsPathTest(PathTest): @@ -3721,6 +3740,31 @@ def check(): env['HOME'] = 'C:\\Users\\eve' check() + def test_from_uri(self): + P = self.cls + # DOS drive paths + self.assertEqual(P.from_uri('file:c:/path/to/file'), P('c:/path/to/file')) + self.assertEqual(P.from_uri('file:c|/path/to/file'), P('c:/path/to/file')) + self.assertEqual(P.from_uri('file:/c|/path/to/file'), P('c:/path/to/file')) + self.assertEqual(P.from_uri('file:///c|/path/to/file'), P('c:/path/to/file')) + # UNC paths + self.assertEqual(P.from_uri('file://server/path/to/file'), P('//server/path/to/file')) + self.assertEqual(P.from_uri('file:////server/path/to/file'), P('//server/path/to/file')) + self.assertEqual(P.from_uri('file://///server/path/to/file'), P('//server/path/to/file')) + # Localhost paths + self.assertEqual(P.from_uri('file://localhost/c:/path/to/file'), P('c:/path/to/file')) + self.assertEqual(P.from_uri('file://localhost/c|/path/to/file'), P('c:/path/to/file')) + # Invalid paths + self.assertRaises(ValueError, P.from_uri, 'foo/bar') + self.assertRaises(ValueError, P.from_uri, 'c:/foo/bar') + self.assertRaises(ValueError, P.from_uri, '//foo/bar') + self.assertRaises(ValueError, P.from_uri, 'file:foo/bar') + self.assertRaises(ValueError, P.from_uri, 'http://foo/bar') + + def test_from_uri_pathname2url(self): + P = self.cls + self.assertEqual(P.from_uri('file:' + pathname2url(r'c:\path\to\file')), P('c:/path/to/file')) + self.assertEqual(P.from_uri('file:' + pathname2url(r'\\server\path\to\file')), P('//server/path/to/file')) class PathSubclassTest(PathTest): diff --git a/Misc/NEWS.d/next/Library/2023-08-04-19-00-53.gh-issue-107465.Vc1Il3.rst b/Misc/NEWS.d/next/Library/2023-08-04-19-00-53.gh-issue-107465.Vc1Il3.rst new file mode 100644 index 0000000000000..e98092f546e39 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-08-04-19-00-53.gh-issue-107465.Vc1Il3.rst @@ -0,0 +1 @@ +Add :meth:`pathlib.Path.from_uri` classmethod.

1 0

gh-110160: Fix flaky `test_find_periodic_pattern` in `string_tests` (#110170)
by vstinner Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/06faa9a39bd93c5e7999d52b52043ecdd0… commit: 06faa9a39bd93c5e7999d52b52043ecdd0774dac branch: main author: Nikita Sobolev <mail(a)sobolevn.me> committer: vstinner <vstinner(a)python.org> date: 2023-10-01T16:20:01+02:00 summary: gh-110160: Fix flaky `test_find_periodic_pattern` in `string_tests` (#110170) files: M Lib/test/string_tests.py diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py index 8d210b198d248..cecf309dca919 100644 --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -327,11 +327,12 @@ def reference_find(p, s): for i in range(len(s)): if s.startswith(p, i): return i + if p == '' and s == '': + return 0 return -1 - rr = random.randrange - choices = random.choices - for _ in range(1000): + def check_pattern(rr): + choices = random.choices p0 = ''.join(choices('abcde', k=rr(10))) * rr(10, 20) p = p0[:len(p0) - rr(10)] # pop off some characters left = ''.join(choices('abcdef', k=rr(2000))) @@ -341,6 +342,13 @@ def reference_find(p, s): self.checkequal(reference_find(p, text), text, 'find', p) + rr = random.randrange + for _ in range(1000): + check_pattern(rr) + + # Test that empty string always work: + check_pattern(lambda *args: 0) + def test_find_many_lengths(self): haystack_repeats = [a * 10**e for e in range(6) for a in (1,2,5)] haystacks = [(n, self.fixtype("abcab"*n + "da")) for n in haystack_repeats]

1 0

PEG generator: bump types-setuptools from 68.1.0.1 to 68.2.0.0 (#110175)
by AlexWaygood Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/038c3564fb4ac6439ff0484e6746b07907… commit: 038c3564fb4ac6439ff0484e6746b0790794f41b branch: main author: dependabot[bot] <49699333+dependabot[bot](a)users.noreply.github.com> committer: AlexWaygood <Alex.Waygood(a)Gmail.com> date: 2023-10-01T12:38:46+01:00 summary: PEG generator: bump types-setuptools from 68.1.0.1 to 68.2.0.0 (#110175) build(deps-dev): bump types-setuptools in /Tools Bumps [types-setuptools](https://github.com/python/typeshed) from 68.1.0.1 to 68.2.0.0. - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-setuptools dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support(a)github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot](a)users.noreply.github.com> files: M Tools/requirements-dev.txt diff --git a/Tools/requirements-dev.txt b/Tools/requirements-dev.txt index 35bceb205e8a9..add28b1bb3818 100644 --- a/Tools/requirements-dev.txt +++ b/Tools/requirements-dev.txt @@ -4,4 +4,4 @@ mypy==1.5.1 # needed for peg_generator: types-psutil==5.9.5.16 -types-setuptools==68.1.0.1 +types-setuptools==68.2.0.0

1 0

gh-110150: Fix base case handling in quantiles() (gh-110151)
by rhettinger Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/62405c7867b03730f0d278ea845855692d… commit: 62405c7867b03730f0d278ea845855692d262d44 branch: main author: Raymond Hettinger <rhettinger(a)users.noreply.github.com> committer: rhettinger <rhettinger(a)users.noreply.github.com> date: 2023-09-30T23:35:54-05:00 summary: gh-110150: Fix base case handling in quantiles() (gh-110151) files: A Misc/NEWS.d/next/Library/2023-09-30-12-50-47.gh-issue-110150.9j0Ij5.rst M Doc/library/statistics.rst M Lib/statistics.py M Lib/test/test_statistics.py diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index f3c1bf20ae3ac..5c8ad3a7dd738 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -585,7 +585,7 @@ However, for reading convenience, most of the examples show sorted sequences. The *data* can be any iterable containing sample data. For meaningful results, the number of data points in *data* should be larger than *n*. - Raises :exc:`StatisticsError` if there are not at least two data points. + Raises :exc:`StatisticsError` if there is not at least one data point. The cut points are linearly interpolated from the two nearest data points. For example, if a cut point falls one-third @@ -625,6 +625,11 @@ However, for reading convenience, most of the examples show sorted sequences. .. versionadded:: 3.8 + .. versionchanged:: 3.13 + No longer raises an exception for an input with only a single data point. + This allows quantile estimates to be built up one sample point + at a time becoming gradually more refined with each new data point. + .. function:: covariance(x, y, /) Return the sample covariance of two inputs *x* and *y*. Covariance diff --git a/Lib/statistics.py b/Lib/statistics.py index 96c803483057e..4da06889c6db4 100644 --- a/Lib/statistics.py +++ b/Lib/statistics.py @@ -844,7 +844,9 @@ def quantiles(data, *, n=4, method='exclusive'): data = sorted(data) ld = len(data) if ld < 2: - raise StatisticsError('must have at least two data points') + if ld == 1: + return data * (n - 1) + raise StatisticsError('must have at least one data point') if method == 'inclusive': m = ld - 1 result = [] diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py index f9b0ac2ad7b11..b24fc3c3d077f 100644 --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -2454,6 +2454,11 @@ def f(x): data = random.choices(range(100), k=k) q1, q2, q3 = quantiles(data, method='inclusive') self.assertEqual(q2, statistics.median(data)) + # Base case with a single data point: When estimating quantiles from + # a sample, we want to be able to add one sample point at a time, + # getting increasingly better estimates. + self.assertEqual(quantiles([10], n=4), [10.0, 10.0, 10.0]) + self.assertEqual(quantiles([10], n=4, method='exclusive'), [10.0, 10.0, 10.0]) def test_equal_inputs(self): quantiles = statistics.quantiles @@ -2504,7 +2509,7 @@ def test_error_cases(self): with self.assertRaises(ValueError): quantiles([10, 20, 30], method='X') # method is unknown with self.assertRaises(StatisticsError): - quantiles([10], n=4) # not enough data points + quantiles([], n=4) # not enough data points with self.assertRaises(TypeError): quantiles([10, None, 30], n=4) # data is non-numeric diff --git a/Misc/NEWS.d/next/Library/2023-09-30-12-50-47.gh-issue-110150.9j0Ij5.rst b/Misc/NEWS.d/next/Library/2023-09-30-12-50-47.gh-issue-110150.9j0Ij5.rst new file mode 100644 index 0000000000000..3c4dde59f71a9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-09-30-12-50-47.gh-issue-110150.9j0Ij5.rst @@ -0,0 +1,2 @@ +Fix base case handling in statistics.quantiles. Now allows a single data +point.

1 0

gh-109649: Use os.process_cpu_count() (#110165)
by vstinner Oct. 1, 2023

Oct. 1, 2023

https://github.com/python/cpython/commit/a46e96076898d126c9f276aef1934195aa… commit: a46e96076898d126c9f276aef1934195aac34b4e branch: main author: Victor Stinner <vstinner(a)python.org> committer: vstinner <vstinner(a)python.org> date: 2023-10-01T03:14:57+02:00 summary: gh-109649: Use os.process_cpu_count() (#110165) Replace os.cpu_count() with os.process_cpu_count() in modules: * compileall * concurrent.futures * multiprocessing Replace os.cpu_count() with os.process_cpu_count() in programs: * _decimal deccheck.py test * freeze.py * multissltests.py * python -m test (regrtest) * wasm_build.py Other changes: * test.pythoninfo logs os.process_cpu_count(). * regrtest gets os.process_cpu_count() / os.cpu_count() in headers. files: A Misc/NEWS.d/next/Library/2023-10-01-01-47-21.gh-issue-109649.BizOaD.rst M Doc/library/compileall.rst M Doc/library/concurrent.futures.rst M Doc/library/multiprocessing.rst M Doc/whatsnew/3.13.rst M Lib/concurrent/futures/process.py M Lib/concurrent/futures/thread.py M Lib/multiprocessing/pool.py M Lib/test/libregrtest/main.py M Lib/test/libregrtest/utils.py M Lib/test/pythoninfo.py M Lib/test/test_concurrent_futures/test_thread_pool.py M Modules/_decimal/tests/deccheck.py M Tools/freeze/test/freeze.py M Tools/ssl/multissltests.py M Tools/wasm/wasm_build.py diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index a7455aeb0ec1c..b4723b98f67bb 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -90,7 +90,7 @@ compile Python sources. .. cmdoption:: -j N Use *N* workers to compile the files within the given directory. - If ``0`` is used, then the result of :func:`os.cpu_count()` + If ``0`` is used, then the result of :func:`os.process_cpu_count()` will be used. .. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash] diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 6503d1fcf70a3..dca51459a2df9 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -188,6 +188,10 @@ And:: ThreadPoolExecutor now reuses idle worker threads before starting *max_workers* worker threads too. + .. versionchanged:: 3.13 + Default value of *max_workers* is changed to + ``min(32, (os.process_cpu_count() or 1) + 4)``. + .. _threadpoolexecutor-example: @@ -243,7 +247,7 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not - given, it will default to the number of processors on the machine. + given, it will default to :func:`os.process_cpu_count`. If *max_workers* is less than or equal to ``0``, then a :exc:`ValueError` will be raised. On Windows, *max_workers* must be less than or equal to ``61``. If it is not @@ -301,6 +305,10 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. different start method. See the :func:`os.fork` documentation for further explanation. + .. versionchanged:: 3.13 + *max_workers* uses :func:`os.process_cpu_count` by default, instead of + :func:`os.cpu_count`. + .. _processpoolexecutor-example: ProcessPoolExecutor Example diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 2f0f1f800fdc9..d19f911dd7016 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -996,13 +996,13 @@ Miscellaneous This number is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with - ``len(os.sched_getaffinity(0))`` + :func:`os.process_cpu_count`. When the number of CPUs cannot be determined a :exc:`NotImplementedError` is raised. .. seealso:: - :func:`os.cpu_count` + :func:`os.cpu_count` and :func:`os.process_cpu_count` .. function:: current_process() @@ -2214,7 +2214,7 @@ with the :class:`Pool` class. callbacks and has a parallel map implementation. *processes* is the number of worker processes to use. If *processes* is - ``None`` then the number returned by :func:`os.cpu_count` is used. + ``None`` then the number returned by :func:`os.process_cpu_count` is used. If *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. @@ -2249,6 +2249,10 @@ with the :class:`Pool` class. .. versionadded:: 3.4 *context* + .. versionchanged:: 3.13 + *processes* uses :func:`os.process_cpu_count` by default, instead of + :func:`os.cpu_count`. + .. note:: Worker processes within a :class:`Pool` typically live for the complete @@ -2775,7 +2779,7 @@ worker threads rather than worker processes. :meth:`~multiprocessing.pool.Pool.terminate` manually. *processes* is the number of worker threads to use. If *processes* is - ``None`` then the number returned by :func:`os.cpu_count` is used. + ``None`` then the number returned by :func:`os.process_cpu_count` is used. If *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst index 484443a086fdd..a789084a79c39 100644 --- a/Doc/whatsnew/3.13.rst +++ b/Doc/whatsnew/3.13.rst @@ -91,6 +91,13 @@ Other Language Changes of the ``optimize`` argument. (Contributed by Irit Katriel in :gh:`108113`). +* :mod:`multiprocessing`, :mod:`concurrent.futures`, :mod:`compileall`: + Replace :func:`os.cpu_count` with :func:`os.process_cpu_count` to select the + default number of worker threads and processes. Get the CPU affinity + if supported. + (Contributed by Victor Stinner in :gh:`109649`.) + + New Modules =========== diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py index 3990e6b1833d7..ffaffdb8b3d0a 100644 --- a/Lib/concurrent/futures/process.py +++ b/Lib/concurrent/futures/process.py @@ -666,7 +666,7 @@ def __init__(self, max_workers=None, mp_context=None, _check_system_limits() if max_workers is None: - self._max_workers = os.cpu_count() or 1 + self._max_workers = os.process_cpu_count() or 1 if sys.platform == 'win32': self._max_workers = min(_MAX_WINDOWS_WORKERS, self._max_workers) diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py index 3b3a36a509333..a024033f35fb5 100644 --- a/Lib/concurrent/futures/thread.py +++ b/Lib/concurrent/futures/thread.py @@ -139,10 +139,10 @@ def __init__(self, max_workers=None, thread_name_prefix='', # * CPU bound task which releases GIL # * I/O bound task (which releases GIL, of course) # - # We use cpu_count + 4 for both types of tasks. + # We use process_cpu_count + 4 for both types of tasks. # But we limit it to 32 to avoid consuming surprisingly large resource # on many core machine. - max_workers = min(32, (os.cpu_count() or 1) + 4) + max_workers = min(32, (os.process_cpu_count() or 1) + 4) if max_workers <= 0: raise ValueError("max_workers must be greater than 0") diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py index 4f5d88cb975cb..f979890170b1a 100644 --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -200,7 +200,7 @@ def __init__(self, processes=None, initializer=None, initargs=(), self._initargs = initargs if processes is None: - processes = os.cpu_count() or 1 + processes = os.process_cpu_count() or 1 if processes < 1: raise ValueError("Number of processes must be at least 1") if maxtasksperchild is not None: diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index 19bf235845603..5f2baac9cba9e 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -426,7 +426,7 @@ def _run_tests(self, selected: TestTuple, tests: TestList | None) -> int: if self.num_workers < 0: # Use all CPUs + 2 extra worker processes for tests # that like to sleep - self.num_workers = (os.cpu_count() or 1) + 2 + self.num_workers = (os.process_cpu_count() or 1) + 2 # For a partial run, we do not need to clutter the output. if (self.want_header diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index d2c274d997073..86fb820a23f53 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -546,6 +546,9 @@ def display_header(use_resources: tuple[str, ...], cpu_count = os.cpu_count() if cpu_count: + process_cpu_count = os.process_cpu_count() + if process_cpu_count and process_cpu_count != cpu_count: + cpu_count = f"{process_cpu_count} (process) / {cpu_count} (system)" print("== CPU count:", cpu_count) print("== encodings: locale=%s, FS=%s" % (locale.getencoding(), sys.getfilesystemencoding())) diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index 0e7528ef97c5f..58d906ffc62a5 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -239,6 +239,7 @@ def format_attr(attr, value): 'getresgid', 'getresuid', 'getuid', + 'process_cpu_count', 'uname', ): call_func(info_add, 'os.%s' % func, os, func) diff --git a/Lib/test/test_concurrent_futures/test_thread_pool.py b/Lib/test/test_concurrent_futures/test_thread_pool.py index 812f989d8f3ad..5926a632aa4be 100644 --- a/Lib/test/test_concurrent_futures/test_thread_pool.py +++ b/Lib/test/test_concurrent_futures/test_thread_pool.py @@ -25,7 +25,7 @@ def record_finished(n): def test_default_workers(self): executor = self.executor_type() - expected = min(32, (os.cpu_count() or 1) + 4) + expected = min(32, (os.process_cpu_count() or 1) + 4) self.assertEqual(executor._max_workers, expected) def test_saturation(self): diff --git a/Misc/NEWS.d/next/Library/2023-10-01-01-47-21.gh-issue-109649.BizOaD.rst b/Misc/NEWS.d/next/Library/2023-10-01-01-47-21.gh-issue-109649.BizOaD.rst new file mode 100644 index 0000000000000..888fd79962b41 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-10-01-01-47-21.gh-issue-109649.BizOaD.rst @@ -0,0 +1,4 @@ +:mod:`multiprocessing`, :mod:`concurrent.futures`, :mod:`compileall`: +Replace :func:`os.cpu_count` with :func:`os.process_cpu_count` to select the +default number of worker threads and processes. Get the CPU affinity if +supported. Patch by Victor Stinner. diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index edf753f3704a1..bf277dd6879ff 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -1301,7 +1301,7 @@ def tfunc(): out, _ = p.communicate() write_output(out, p.returncode) - N = os.cpu_count() + N = os.process_cpu_count() t = N * [None] for i in range(N): diff --git a/Tools/freeze/test/freeze.py b/Tools/freeze/test/freeze.py index cdf77c57bbb6a..9030ad4d4e5f9 100644 --- a/Tools/freeze/test/freeze.py +++ b/Tools/freeze/test/freeze.py @@ -130,7 +130,7 @@ def prepare(script=None, outdir=None): if not MAKE: raise UnsupportedError('make') - cores = os.cpu_count() + cores = os.process_cpu_count() if cores and cores >= 3: # this test is most often run as part of the whole suite with a lot # of other tests running in parallel, from 1-2 vCPU systems up to diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py index f066fb52cfd49..120e3883adc79 100755 --- a/Tools/ssl/multissltests.py +++ b/Tools/ssl/multissltests.py @@ -151,7 +151,10 @@ class AbstractBuilder(object): build_template = None depend_target = None install_target = 'install' - jobs = os.cpu_count() + if hasattr(os, 'process_cpu_count'): + jobs = os.process_cpu_count() + else: + jobs = os.cpu_count() module_files = ( os.path.join(PYTHONROOT, "Modules/_ssl.c"), diff --git a/Tools/wasm/wasm_build.py b/Tools/wasm/wasm_build.py index 3558ecd869dfc..c0b9999a5dad0 100755 --- a/Tools/wasm/wasm_build.py +++ b/Tools/wasm/wasm_build.py @@ -516,7 +516,11 @@ def make_cmd(self) -> List[str]: def getenv(self) -> Dict[str, Any]: """Generate environ dict for platform""" env = os.environ.copy() - env.setdefault("MAKEFLAGS", f"-j{os.cpu_count()}") + if hasattr(os, 'process_cpu_count'): + cpu_count = os.process_cpu_count() + else: + cpu_count = os.cpu_count() + env.setdefault("MAKEFLAGS", f"-j{cpu_count}") platenv = self.host.platform.getenv(self) for key, value in platenv.items(): if value is None:

1 0