Mailman 3 March 2024 - Python-checkins

Doc: printf-style library/stdtype improvements (#16741)
by erlend-aasland 31 Mar '24

31 Mar '24

https://github.com/python/cpython/commit/56e99307c49adcc6df355f8070229371c9… commit: 56e99307c49adcc6df355f8070229371c97d654f branch: main author: Adorilson Bezerra <adorilson(a)gmail.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2024-03-31T22:34:54Z summary: Doc: printf-style library/stdtype improvements (#16741) files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index c963519f164dd2..62fc10997fc5b5 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2339,7 +2339,13 @@ String objects have one unique built-in operation: the ``%`` operator (modulo). This is also known as the string *formatting* or *interpolation* operator. Given ``format % values`` (where *format* is a string), ``%`` conversion specifications in *format* are replaced with zero or more elements of *values*. -The effect is similar to using the :c:func:`sprintf` in the C language. +The effect is similar to using the :c:func:`sprintf` function in the C language. +For example: + +.. doctest:: + + >>> print('%s has %d quote types.' % ('Python', 2)) + Python has 2 quote types. If *format* requires a single argument, *values* may be a single non-tuple object. [5]_ Otherwise, *values* must be a tuple with exactly the number of

1 0

[3.12] gh-117387 Remove hash mark from introductory text (GH-117409) (#gh-
by rhettinger 31 Mar '24

31 Mar '24

https://github.com/python/cpython/commit/b912d2b24a3fbd552595e3584bf851f9a2… commit: b912d2b24a3fbd552595e3584bf851f9a2c75127 branch: 3.12 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: rhettinger <rhettinger(a)users.noreply.github.com> date: 2024-03-31T21:24:28Z summary: [3.12] gh-117387 Remove hash mark from introductory text (GH-117409) (#gh- files: M Lib/collections/__init__.py diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index 0e7971a872e3a2..5f000b5f2c38c3 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -638,7 +638,8 @@ def elements(self): >>> sorted(c.elements()) ['A', 'A', 'B', 'B', 'C', 'C'] - # Knuth's example for prime factors of 1836: 2**2 * 3**3 * 17**1 + Knuth's example for prime factors of 1836: 2**2 * 3**3 * 17**1 + >>> import math >>> prime_factors = Counter({2: 2, 3: 3, 17: 1}) >>> math.prod(prime_factors.elements())

1 0

gh-117387 Remove hash mark from introductory text (#117409)
by rhettinger 31 Mar '24

31 Mar '24

https://github.com/python/cpython/commit/18e12641a61a88f7d08b2114ebe965892c… commit: 18e12641a61a88f7d08b2114ebe965892c6661c5 branch: main author: Raymond Hettinger <rhettinger(a)users.noreply.github.com> committer: rhettinger <rhettinger(a)users.noreply.github.com> date: 2024-03-31T16:09:22-05:00 summary: gh-117387 Remove hash mark from introductory text (#117409) files: M Lib/collections/__init__.py diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index 0aa0c3e15e9519..2a35989ee25a5e 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -639,7 +639,8 @@ def elements(self): >>> sorted(c.elements()) ['A', 'A', 'B', 'B', 'C', 'C'] - # Knuth's example for prime factors of 1836: 2**2 * 3**3 * 17**1 + Knuth's example for prime factors of 1836: 2**2 * 3**3 * 17**1 + >>> import math >>> prime_factors = Counter({2: 2, 3: 3, 17: 1}) >>> math.prod(prime_factors.elements())

1 0

gh-102190: Add additional zipfile `pwd=` arg docstrings (gh-102195)
by gpshead 31 Mar '24

31 Mar '24

https://github.com/python/cpython/commit/a32d6939486d7f90ee57e215077f6116e1… commit: a32d6939486d7f90ee57e215077f6116e19de24d branch: main author: Deborah <32307299+dlwrnc(a)users.noreply.github.com> committer: gpshead <greg(a)krypto.org> date: 2024-03-31T20:11:48Z summary: gh-102190: Add additional zipfile `pwd=` arg docstrings (gh-102195) This just documents the parameter that already exists. --------- Co-authored-by: Gregory P. Smith <greg(a)krypto.org> Co-authored-by: Erlend E. Aasland <erlend.aasland(a)protonmail.com> files: M Lib/zipfile/__init__.py diff --git a/Lib/zipfile/__init__.py b/Lib/zipfile/__init__.py index b330ece4b276ad..e4603b559f5962 100644 --- a/Lib/zipfile/__init__.py +++ b/Lib/zipfile/__init__.py @@ -1585,7 +1585,8 @@ def comment(self, comment): self._didModify = True def read(self, name, pwd=None): - """Return file bytes for name.""" + """Return file bytes for name. 'pwd' is the password to decrypt + encrypted files.""" with self.open(name, "r", pwd) as fp: return fp.read() @@ -1737,7 +1738,8 @@ def extract(self, member, path=None, pwd=None): """Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. `member' may be a filename or a ZipInfo object. You can - specify a different directory using `path'. + specify a different directory using `path'. You can specify the + password to decrypt the file using 'pwd'. """ if path is None: path = os.getcwd() @@ -1750,7 +1752,8 @@ def extractall(self, path=None, members=None, pwd=None): """Extract all members from the archive to the current working directory. `path' specifies a different directory to extract to. `members' is optional and must be a subset of the list returned - by namelist(). + by namelist(). You can specify the password to decrypt all files + using 'pwd'. """ if members is None: members = self.namelist()

1 0

Link to the Python type system specification (#117400)
by hauntsaninja 31 Mar '24

31 Mar '24

https://github.com/python/cpython/commit/262445358e21c56d7c68e3ee76c13e469d… commit: 262445358e21c56d7c68e3ee76c13e469d2ea348 branch: main author: Shantanu <12621235+hauntsaninja(a)users.noreply.github.com> committer: hauntsaninja <12621235+hauntsaninja(a)users.noreply.github.com> date: 2024-03-31T12:02:48-07:00 summary: Link to the Python type system specification (#117400) files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 713ad1c83546d1..73214e18d556b2 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -23,27 +23,25 @@ -------------- -This module provides runtime support for type hints. For the original -specification of the typing system, see :pep:`484`. For a simplified -introduction to type hints, see :pep:`483`. +This module provides runtime support for type hints. +Consider the function below:: -The function below takes and returns a string and is annotated as follows:: + def moon_weight(earth_weight: float) -> str: + return f'On the moon, you would weigh {earth_weight * 0.166} kilograms.' - def greeting(name: str) -> str: - return 'Hello ' + name +The function ``moon_weight`` takes an argument expected to be an instance of :class:`float`, +as indicated by the *type hint* ``earth_weight: float``. The function is expected to +return an instance of :class:`str`, as indicated by the ``-> str`` hint. -In the function ``greeting``, the argument ``name`` is expected to be of type -:class:`str` and the return type :class:`str`. Subtypes are accepted as -arguments. +While type hints can be simple classes like :class:`float` or :class:`str`, +they can also be more complex. The :mod:`typing` module provides a vocabulary of +more advanced type hints. New features are frequently added to the ``typing`` module. The `typing_extensions <https://pypi.org/project/typing-extensions/>`_ package provides backports of these new features to older versions of Python. -For a summary of deprecated features and a deprecation timeline, please see -`Deprecation Timeline of Major Features`_. - .. seealso:: `"Typing cheat sheet" <https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html>`_ @@ -61,67 +59,11 @@ For a summary of deprecated features and a deprecation timeline, please see .. _relevant-peps: -Relevant PEPs -============= - -Since the initial introduction of type hints in :pep:`484` and :pep:`483`, a -number of PEPs have modified and enhanced Python's framework for type -annotations: - -.. raw:: html - - <details> - <summary><a style="cursor:pointer;">The full list of PEPs</a></summary> - -* :pep:`526`: Syntax for Variable Annotations - *Introducing* syntax for annotating variables outside of function - definitions, and :data:`ClassVar` -* :pep:`544`: Protocols: Structural subtyping (static duck typing) - *Introducing* :class:`Protocol` and the - :func:`@runtime_checkable<runtime_checkable>` decorator -* :pep:`585`: Type Hinting Generics In Standard Collections - *Introducing* :class:`types.GenericAlias` and the ability to use standard - library classes as :ref:`generic types<types-genericalias>` -* :pep:`586`: Literal Types - *Introducing* :data:`Literal` -* :pep:`589`: TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys - *Introducing* :class:`TypedDict` -* :pep:`591`: Adding a final qualifier to typing - *Introducing* :data:`Final` and the :func:`@final<final>` decorator -* :pep:`593`: Flexible function and variable annotations - *Introducing* :data:`Annotated` -* :pep:`604`: Allow writing union types as ``X | Y`` - *Introducing* :data:`types.UnionType` and the ability to use - the binary-or operator ``|`` to signify a - :ref:`union of types<types-union>` -* :pep:`612`: Parameter Specification Variables - *Introducing* :class:`ParamSpec` and :data:`Concatenate` -* :pep:`613`: Explicit Type Aliases - *Introducing* :data:`TypeAlias` -* :pep:`646`: Variadic Generics - *Introducing* :data:`TypeVarTuple` -* :pep:`647`: User-Defined Type Guards - *Introducing* :data:`TypeGuard` -* :pep:`655`: Marking individual TypedDict items as required or potentially missing - *Introducing* :data:`Required` and :data:`NotRequired` -* :pep:`673`: Self type - *Introducing* :data:`Self` -* :pep:`675`: Arbitrary Literal String Type - *Introducing* :data:`LiteralString` -* :pep:`681`: Data Class Transforms - *Introducing* the :func:`@dataclass_transform<dataclass_transform>` decorator -* :pep:`692`: Using ``TypedDict`` for more precise ``**kwargs`` typing - *Introducing* a new way of typing ``**kwargs`` with :data:`Unpack` and - :data:`TypedDict` -* :pep:`695`: Type Parameter Syntax - *Introducing* builtin syntax for creating generic functions, classes, and type aliases. -* :pep:`698`: Adding an override decorator to typing - *Introducing* the :func:`@override<override>` decorator - -.. raw:: html - - </details> - <br> +Specification for the Python Type System +======================================== + +The canonical, up-to-date specification of the Python type system can be +found at `"Specification for the Python type system" <https://typing.readthedocs.io/en/latest/spec/index.html>`_. .. _type-aliases:

1 0

GH-114575: Rename `PurePath.pathmod` to `PurePath.parser` (#116513)
by barneygale 31 Mar '24

31 Mar '24

https://github.com/python/cpython/commit/752e18389ed03087b51b38eac9769ef8df… commit: 752e18389ed03087b51b38eac9769ef8dfd167b7 branch: main author: Barney Gale <barney.gale(a)gmail.com> committer: barneygale <barney.gale(a)gmail.com> date: 2024-03-31T19:14:48+01:00 summary: GH-114575: Rename `PurePath.pathmod` to `PurePath.parser` (#116513) And rename the private base class from `PathModuleBase` to `ParserBase`. files: M Doc/library/pathlib.rst M Doc/whatsnew/3.13.rst M Lib/pathlib/__init__.py M Lib/pathlib/_abc.py M Lib/test/test_pathlib/test_pathlib.py M Lib/test/test_pathlib/test_pathlib_abc.py diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 4fba3622b073a7..9122df7a476632 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -303,10 +303,10 @@ Methods and properties Pure paths provide the following methods and properties: -.. attribute:: PurePath.pathmod +.. attribute:: PurePath.parser The implementation of the :mod:`os.path` module used for low-level path - operations: either :mod:`posixpath` or :mod:`ntpath`. + parsing and joining: either :mod:`posixpath` or :mod:`ntpath`. .. versionadded:: 3.13 diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst index f50364a7ddcc2a..45f7f50bf9f46b 100644 --- a/Doc/whatsnew/3.13.rst +++ b/Doc/whatsnew/3.13.rst @@ -524,6 +524,10 @@ pathlib shell-style wildcards, including the recursive wildcard "``**``". (Contributed by Barney Gale in :gh:`73435`.) +* Add :attr:`pathlib.PurePath.parser` class attribute that stores the + implementation of :mod:`os.path` used for low-level path parsing and + joining: either ``posixpath`` or ``ntpath``. + * Add *follow_symlinks* keyword-only argument to :meth:`pathlib.Path.glob`, :meth:`~pathlib.Path.rglob`, :meth:`~pathlib.Path.is_file`, :meth:`~pathlib.Path.is_dir`, :meth:`~pathlib.Path.owner`, diff --git a/Lib/pathlib/__init__.py b/Lib/pathlib/__init__.py index 46834b1a76a6eb..6cccfb864e8206 100644 --- a/Lib/pathlib/__init__.py +++ b/Lib/pathlib/__init__.py @@ -110,7 +110,7 @@ class PurePath(_abc.PurePathBase): # path. It's set when `__hash__()` is called for the first time. '_hash', ) - pathmod = os.path + parser = os.path def __new__(cls, *args, **kwargs): """Construct a PurePath from one or several strings and or existing @@ -126,7 +126,7 @@ def __init__(self, *args): paths = [] for arg in args: if isinstance(arg, PurePath): - if arg.pathmod is ntpath and self.pathmod is posixpath: + if arg.parser is ntpath and self.parser is posixpath: # GH-103631: Convert separators for backwards compatibility. paths.extend(path.replace('\\', '/') for path in arg._raw_paths) else: @@ -187,7 +187,7 @@ def _str_normcase(self): try: return self._str_normcase_cached except AttributeError: - if _abc._is_case_sensitive(self.pathmod): + if _abc._is_case_sensitive(self.parser): self._str_normcase_cached = str(self) else: self._str_normcase_cached = str(self).lower() @@ -203,7 +203,7 @@ def __hash__(self): def __eq__(self, other): if not isinstance(other, PurePath): return NotImplemented - return self._str_normcase == other._str_normcase and self.pathmod is other.pathmod + return self._str_normcase == other._str_normcase and self.parser is other.parser @property def _parts_normcase(self): @@ -211,26 +211,26 @@ def _parts_normcase(self): try: return self._parts_normcase_cached except AttributeError: - self._parts_normcase_cached = self._str_normcase.split(self.pathmod.sep) + self._parts_normcase_cached = self._str_normcase.split(self.parser.sep) return self._parts_normcase_cached def __lt__(self, other): - if not isinstance(other, PurePath) or self.pathmod is not other.pathmod: + if not isinstance(other, PurePath) or self.parser is not other.parser: return NotImplemented return self._parts_normcase < other._parts_normcase def __le__(self, other): - if not isinstance(other, PurePath) or self.pathmod is not other.pathmod: + if not isinstance(other, PurePath) or self.parser is not other.parser: return NotImplemented return self._parts_normcase <= other._parts_normcase def __gt__(self, other): - if not isinstance(other, PurePath) or self.pathmod is not other.pathmod: + if not isinstance(other, PurePath) or self.parser is not other.parser: return NotImplemented return self._parts_normcase > other._parts_normcase def __ge__(self, other): - if not isinstance(other, PurePath) or self.pathmod is not other.pathmod: + if not isinstance(other, PurePath) or self.parser is not other.parser: return NotImplemented return self._parts_normcase >= other._parts_normcase @@ -247,10 +247,10 @@ def __str__(self): @classmethod def _format_parsed_parts(cls, drv, root, tail): if drv or root: - return drv + root + cls.pathmod.sep.join(tail) - elif tail and cls.pathmod.splitdrive(tail[0])[0]: + return drv + root + cls.parser.sep.join(tail) + elif tail and cls.parser.splitdrive(tail[0])[0]: tail = ['.'] + tail - return cls.pathmod.sep.join(tail) + return cls.parser.sep.join(tail) def _from_parsed_parts(self, drv, root, tail): path_str = self._format_parsed_parts(drv, root, tail) @@ -265,11 +265,11 @@ def _from_parsed_parts(self, drv, root, tail): def _parse_path(cls, path): if not path: return '', '', [] - sep = cls.pathmod.sep - altsep = cls.pathmod.altsep + sep = cls.parser.sep + altsep = cls.parser.altsep if altsep: path = path.replace(altsep, sep) - drv, root, rel = cls.pathmod.splitroot(path) + drv, root, rel = cls.parser.splitroot(path) if not root and drv.startswith(sep) and not drv.endswith(sep): drv_parts = drv.split(sep) if len(drv_parts) == 4 and drv_parts[2] not in '?.': @@ -290,7 +290,7 @@ def _raw_path(self): elif len(paths) == 1: path = paths[0] else: - path = self.pathmod.join(*paths) + path = self.parser.join(*paths) return path @property @@ -360,8 +360,8 @@ def name(self): def with_name(self, name): """Return a new path with the file name changed.""" - m = self.pathmod - if not name or m.sep in name or (m.altsep and m.altsep in name) or name == '.': + p = self.parser + if not name or p.sep in name or (p.altsep and p.altsep in name) or name == '.': raise ValueError(f"Invalid name {name!r}") tail = self._tail.copy() if not tail: @@ -413,13 +413,13 @@ def is_relative_to(self, other, /, *_deprecated): def is_absolute(self): """True if the path is absolute (has both a root and, if applicable, a drive).""" - if self.pathmod is posixpath: + if self.parser is posixpath: # Optimization: work with raw paths on POSIX. for path in self._raw_paths: if path.startswith('/'): return True return False - return self.pathmod.isabs(self) + return self.parser.isabs(self) def is_reserved(self): """Return True if the path contains one of the special names reserved @@ -428,8 +428,8 @@ def is_reserved(self): "for removal in Python 3.15. Use os.path.isreserved() to " "detect reserved paths on Windows.") warnings.warn(msg, DeprecationWarning, stacklevel=2) - if self.pathmod is ntpath: - return self.pathmod.isreserved(self) + if self.parser is ntpath: + return self.parser.isreserved(self) return False def as_uri(self): @@ -462,7 +462,7 @@ def _pattern_stack(self): raise NotImplementedError("Non-relative patterns are unsupported") elif not parts: raise ValueError("Unacceptable pattern: {!r}".format(pattern)) - elif pattern[-1] in (self.pathmod.sep, self.pathmod.altsep): + elif pattern[-1] in (self.parser.sep, self.parser.altsep): # GH-65238: pathlib doesn't preserve trailing slash. Add it back. parts.append('') parts.reverse() @@ -487,7 +487,7 @@ class PurePosixPath(PurePath): On a POSIX system, instantiating a PurePath should return this object. However, you can also instantiate it directly on any system. """ - pathmod = posixpath + parser = posixpath __slots__ = () @@ -497,7 +497,7 @@ class PureWindowsPath(PurePath): On a Windows system, instantiating a PurePath should return this object. However, you can also instantiate it directly on any system. """ - pathmod = ntpath + parser = ntpath __slots__ = () @@ -607,7 +607,7 @@ def _make_child_relpath(self, name): path_str = str(self) tail = self._tail if tail: - path_str = f'{path_str}{self.pathmod.sep}{name}' + path_str = f'{path_str}{self.parser.sep}{name}' elif path_str != '.': path_str = f'{path_str}{name}' else: @@ -675,7 +675,7 @@ def absolute(self): drive, root, rel = os.path.splitroot(cwd) if not rel: return self._from_parsed_parts(drive, root, self._tail) - tail = rel.split(self.pathmod.sep) + tail = rel.split(self.parser.sep) tail.extend(self._tail) return self._from_parsed_parts(drive, root, tail) diff --git a/Lib/pathlib/_abc.py b/Lib/pathlib/_abc.py index 645d62a0f0699a..932020e6d0866c 100644 --- a/Lib/pathlib/_abc.py +++ b/Lib/pathlib/_abc.py @@ -37,8 +37,8 @@ def _ignore_error(exception): @functools.cache -def _is_case_sensitive(pathmod): - return pathmod.normcase('Aa') == 'Aa' +def _is_case_sensitive(parser): + return parser.normcase('Aa') == 'Aa' # # Globbing helpers @@ -156,12 +156,12 @@ class UnsupportedOperation(NotImplementedError): pass -class PathModuleBase: - """Base class for path modules, which do low-level path manipulation. +class ParserBase: + """Base class for path parsers, which do low-level path manipulation. - Path modules provide a subset of the os.path API, specifically those + Path parsers provide a subset of the os.path API, specifically those functions needed to provide PurePathBase functionality. Each PurePathBase - subclass references its path module via a 'pathmod' class attribute. + subclass references its path parser via a 'parser' class attribute. Every method in this base class raises an UnsupportedOperation exception. """ @@ -221,10 +221,10 @@ class PurePathBase: # work from occurring when `resolve()` calls `stat()` or `readlink()`. '_resolving', ) - pathmod = PathModuleBase() + parser = ParserBase() def __init__(self, path, *paths): - self._raw_path = self.pathmod.join(path, *paths) if paths else path + self._raw_path = self.parser.join(path, *paths) if paths else path if not isinstance(self._raw_path, str): raise TypeError( f"path should be a str, not {type(self._raw_path).__name__!r}") @@ -245,17 +245,17 @@ def __str__(self): def as_posix(self): """Return the string representation of the path with forward (/) slashes.""" - return str(self).replace(self.pathmod.sep, '/') + return str(self).replace(self.parser.sep, '/') @property def drive(self): """The drive prefix (letter or UNC path), if any.""" - return self.pathmod.splitdrive(self.anchor)[0] + return self.parser.splitdrive(self.anchor)[0] @property def root(self): """The root of the path, if any.""" - return self.pathmod.splitdrive(self.anchor)[1] + return self.parser.splitdrive(self.anchor)[1] @property def anchor(self): @@ -265,7 +265,7 @@ def anchor(self): @property def name(self): """The final path component, if any.""" - return self.pathmod.split(self._raw_path)[1] + return self.parser.split(self._raw_path)[1] @property def suffix(self): @@ -306,7 +306,7 @@ def stem(self): def with_name(self, name): """Return a new path with the file name changed.""" - split = self.pathmod.split + split = self.parser.split if split(name)[0]: raise ValueError(f"Invalid name {name!r}") return self.with_segments(split(self._raw_path)[0], name) @@ -419,7 +419,7 @@ def _stack(self): uppermost parent of the path (equivalent to path.parents[-1]), and *parts* is a reversed list of parts following the anchor. """ - split = self.pathmod.split + split = self.parser.split path = self._raw_path parent, name = split(path) names = [] @@ -433,7 +433,7 @@ def _stack(self): def parent(self): """The logical parent of the path.""" path = self._raw_path - parent = self.pathmod.split(path)[0] + parent = self.parser.split(path)[0] if path != parent: parent = self.with_segments(parent) parent._resolving = self._resolving @@ -443,7 +443,7 @@ def parent(self): @property def parents(self): """A sequence of this path's logical parents.""" - split = self.pathmod.split + split = self.parser.split path = self._raw_path parent = split(path)[0] parents = [] @@ -456,7 +456,7 @@ def parents(self): def is_absolute(self): """True if the path is absolute (has both a root and, if applicable, a drive).""" - return self.pathmod.isabs(self._raw_path) + return self.parser.isabs(self._raw_path) @property def _pattern_stack(self): @@ -481,8 +481,8 @@ def match(self, path_pattern, *, case_sensitive=None): if not isinstance(path_pattern, PurePathBase): path_pattern = self.with_segments(path_pattern) if case_sensitive is None: - case_sensitive = _is_case_sensitive(self.pathmod) - sep = path_pattern.pathmod.sep + case_sensitive = _is_case_sensitive(self.parser) + sep = path_pattern.parser.sep path_parts = self.parts[::-1] pattern_parts = path_pattern.parts[::-1] if not pattern_parts: @@ -505,8 +505,8 @@ def full_match(self, pattern, *, case_sensitive=None): if not isinstance(pattern, PurePathBase): pattern = self.with_segments(pattern) if case_sensitive is None: - case_sensitive = _is_case_sensitive(self.pathmod) - match = _compile_pattern(pattern._pattern_str, pattern.pathmod.sep, case_sensitive) + case_sensitive = _is_case_sensitive(self.parser) + match = _compile_pattern(pattern._pattern_str, pattern.parser.sep, case_sensitive) return match(self._pattern_str) is not None @@ -797,12 +797,12 @@ def glob(self, pattern, *, case_sensitive=None, follow_symlinks=True): pattern = self.with_segments(pattern) if case_sensitive is None: # TODO: evaluate case-sensitivity of each directory in _select_children(). - case_sensitive = _is_case_sensitive(self.pathmod) + case_sensitive = _is_case_sensitive(self.parser) stack = pattern._pattern_stack specials = ('', '.', '..') deduplicate_paths = False - sep = self.pathmod.sep + sep = self.parser.sep paths = iter([self] if self.is_dir() else []) while stack: part = stack.pop() @@ -973,7 +973,7 @@ def resolve(self, strict=False): continue path_tail.append(part) if querying and part != '..': - path = self.with_segments(path_root + self.pathmod.sep.join(path_tail)) + path = self.with_segments(path_root + self.parser.sep.join(path_tail)) path._resolving = True try: st = path.stat(follow_symlinks=False) @@ -1002,7 +1002,7 @@ def resolve(self, strict=False): raise else: querying = False - return self.with_segments(path_root + self.pathmod.sep.join(path_tail)) + return self.with_segments(path_root + self.parser.sep.join(path_tail)) def symlink_to(self, target, target_is_directory=False): """ diff --git a/Lib/test/test_pathlib/test_pathlib.py b/Lib/test/test_pathlib/test_pathlib.py index 3a6f73c4fe82a4..651d66656cbd61 100644 --- a/Lib/test/test_pathlib/test_pathlib.py +++ b/Lib/test/test_pathlib/test_pathlib.py @@ -65,7 +65,7 @@ def test_concrete_class(self): p = self.cls('a') self.assertIs(type(p), expected) - def test_concrete_pathmod(self): + def test_concrete_parser(self): if self.cls is pathlib.PurePosixPath: expected = posixpath elif self.cls is pathlib.PureWindowsPath: @@ -73,19 +73,19 @@ def test_concrete_pathmod(self): else: expected = os.path p = self.cls('a') - self.assertIs(p.pathmod, expected) + self.assertIs(p.parser, expected) - def test_different_pathmods_unequal(self): + def test_different_parsers_unequal(self): p = self.cls('a') - if p.pathmod is posixpath: + if p.parser is posixpath: q = pathlib.PureWindowsPath('a') else: q = pathlib.PurePosixPath('a') self.assertNotEqual(p, q) - def test_different_pathmods_unordered(self): + def test_different_parsers_unordered(self): p = self.cls('a') - if p.pathmod is posixpath: + if p.parser is posixpath: q = pathlib.PureWindowsPath('a') else: q = pathlib.PurePosixPath('a') @@ -108,16 +108,16 @@ def test_constructor_nested(self): self.assertEqual(P(P('./a:b')), P('./a:b')) def _check_parse_path(self, raw_path, *expected): - sep = self.pathmod.sep + sep = self.parser.sep actual = self.cls._parse_path(raw_path.replace('/', sep)) self.assertEqual(actual, expected) - if altsep := self.pathmod.altsep: + if altsep := self.parser.altsep: actual = self.cls._parse_path(raw_path.replace('/', altsep)) self.assertEqual(actual, expected) def test_parse_path_common(self): check = self._check_parse_path - sep = self.pathmod.sep + sep = self.parser.sep check('', '', '', []) check('a', '', '', ['a']) check('a/', '', '', ['a']) @@ -523,10 +523,10 @@ class PathTest(test_pathlib_abc.DummyPathTest, PurePathTest): def setUp(self): super().setUp() - os.chmod(self.pathmod.join(self.base, 'dirE'), 0) + os.chmod(self.parser.join(self.base, 'dirE'), 0) def tearDown(self): - os.chmod(self.pathmod.join(self.base, 'dirE'), 0o777) + os.chmod(self.parser.join(self.base, 'dirE'), 0o777) os_helper.rmtree(self.base) def tempdir(self): @@ -541,8 +541,8 @@ def test_matches_pathbase_api(self): path_names = {name for name in dir(pathlib._abc.PathBase) if name[0] != '_'} self.assertEqual(our_names, path_names) for attr_name in our_names: - if attr_name == 'pathmod': - # On Windows, Path.pathmod is ntpath, but PathBase.pathmod is + if attr_name == 'parser': + # On Windows, Path.parser is ntpath, but PathBase.parser is # posixpath, and so their docstrings differ. continue our_attr = getattr(self.cls, attr_name) @@ -557,9 +557,9 @@ def test_concrete_class(self): p = self.cls('a') self.assertIs(type(p), expected) - def test_unsupported_pathmod(self): - if self.cls.pathmod is os.path: - self.skipTest("path flavour is supported") + def test_unsupported_parser(self): + if self.cls.parser is os.path: + self.skipTest("path parser is supported") else: self.assertRaises(pathlib.UnsupportedOperation, self.cls) @@ -809,7 +809,7 @@ def test_hardlink_to(self): self.assertTrue(target.exists()) # Linking to a str of a relative path. link2 = P / 'dirA' / 'fileAAA' - target2 = self.pathmod.join(TESTFN, 'fileA') + target2 = self.parser.join(TESTFN, 'fileA') link2.hardlink_to(target2) self.assertEqual(os.stat(target2).st_size, size) self.assertTrue(link2.exists()) @@ -834,7 +834,7 @@ def test_rename(self): self.assertEqual(q.stat().st_size, size) self.assertFileNotFound(p.stat) # Renaming to a str of a relative path. - r = self.pathmod.join(TESTFN, 'fileAAA') + r = self.parser.join(TESTFN, 'fileAAA') renamed_q = q.rename(r) self.assertEqual(renamed_q, self.cls(r)) self.assertEqual(os.stat(r).st_size, size) @@ -851,7 +851,7 @@ def test_replace(self): self.assertEqual(q.stat().st_size, size) self.assertFileNotFound(p.stat) # Replacing another (existing) path. - r = self.pathmod.join(TESTFN, 'dirB', 'fileB') + r = self.parser.join(TESTFN, 'dirB', 'fileB') replaced_q = q.replace(r) self.assertEqual(replaced_q, self.cls(r)) self.assertEqual(os.stat(r).st_size, size) @@ -1060,9 +1060,9 @@ def test_symlink_to_unsupported(self): def test_is_junction(self): P = self.cls(self.base) - with mock.patch.object(P.pathmod, 'isjunction'): - self.assertEqual(P.is_junction(), P.pathmod.isjunction.return_value) - P.pathmod.isjunction.assert_called_once_with(P) + with mock.patch.object(P.parser, 'isjunction'): + self.assertEqual(P.is_junction(), P.parser.isjunction.return_value) + P.parser.isjunction.assert_called_once_with(P) @unittest.skipUnless(hasattr(os, "mkfifo"), "os.mkfifo() required") @unittest.skipIf(sys.platform == "vxworks", @@ -1294,12 +1294,12 @@ def test_open_mode(self): p = self.cls(self.base) with (p / 'new_file').open('wb'): pass - st = os.stat(self.pathmod.join(self.base, 'new_file')) + st = os.stat(self.parser.join(self.base, 'new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o666) os.umask(0o022) with (p / 'other_new_file').open('wb'): pass - st = os.stat(self.pathmod.join(self.base, 'other_new_file')) + st = os.stat(self.parser.join(self.base, 'other_new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o644) @needs_posix @@ -1322,14 +1322,14 @@ def test_touch_mode(self): self.addCleanup(os.umask, old_mask) p = self.cls(self.base) (p / 'new_file').touch() - st = os.stat(self.pathmod.join(self.base, 'new_file')) + st = os.stat(self.parser.join(self.base, 'new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o666) os.umask(0o022) (p / 'other_new_file').touch() - st = os.stat(self.pathmod.join(self.base, 'other_new_file')) + st = os.stat(self.parser.join(self.base, 'other_new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o644) (p / 'masked_new_file').touch(mode=0o750) - st = os.stat(self.pathmod.join(self.base, 'masked_new_file')) + st = os.stat(self.parser.join(self.base, 'masked_new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o750) @unittest.skipUnless(hasattr(pwd, 'getpwall'), diff --git a/Lib/test/test_pathlib/test_pathlib_abc.py b/Lib/test/test_pathlib/test_pathlib_abc.py index 840fb903fd5338..a7e35a3e1fc7da 100644 --- a/Lib/test/test_pathlib/test_pathlib_abc.py +++ b/Lib/test/test_pathlib/test_pathlib_abc.py @@ -5,7 +5,7 @@ import stat import unittest -from pathlib._abc import UnsupportedOperation, PathModuleBase, PurePathBase, PathBase +from pathlib._abc import UnsupportedOperation, ParserBase, PurePathBase, PathBase import posixpath from test.support.os_helper import TESTFN @@ -38,8 +38,8 @@ def test_is_notimplemented(self): self.assertTrue(isinstance(UnsupportedOperation(), NotImplementedError)) -class PathModuleBaseTest(unittest.TestCase): - cls = PathModuleBase +class ParserBaseTest(unittest.TestCase): + cls = ParserBase def test_unsupported_operation(self): m = self.cls() @@ -109,13 +109,13 @@ def test_magic_methods(self): self.assertIs(P.__gt__, object.__gt__) self.assertIs(P.__ge__, object.__ge__) - def test_pathmod(self): - self.assertIsInstance(self.cls.pathmod, PathModuleBase) + def test_parser(self): + self.assertIsInstance(self.cls.parser, ParserBase) class DummyPurePath(PurePathBase): __slots__ = () - pathmod = posixpath + parser = posixpath def __eq__(self, other): if not isinstance(other, DummyPurePath): @@ -137,14 +137,14 @@ class DummyPurePathTest(unittest.TestCase): def setUp(self): name = self.id().split('.')[-1] - if name in _tests_needing_posix and self.cls.pathmod is not posixpath: + if name in _tests_needing_posix and self.cls.parser is not posixpath: self.skipTest('requires POSIX-flavoured path class') - if name in _tests_needing_windows and self.cls.pathmod is posixpath: + if name in _tests_needing_windows and self.cls.parser is posixpath: self.skipTest('requires Windows-flavoured path class') p = self.cls('a') - self.pathmod = p.pathmod - self.sep = self.pathmod.sep - self.altsep = self.pathmod.altsep + self.parser = p.parser + self.sep = self.parser.sep + self.altsep = self.parser.altsep def test_constructor_common(self): P = self.cls @@ -1411,7 +1411,7 @@ class DummyPath(PathBase): memory. """ __slots__ = () - pathmod = posixpath + parser = posixpath _files = {} _directories = {} @@ -1530,7 +1530,7 @@ def setUp(self): name = self.id().split('.')[-1] if name in _tests_needing_symlinks and not self.can_symlink: self.skipTest('requires symlinks') - pathmod = self.cls.pathmod + parser = self.cls.parser p = self.cls(self.base) p.mkdir(parents=True) p.joinpath('dirA').mkdir() @@ -1552,8 +1552,8 @@ def setUp(self): p.joinpath('linkA').symlink_to('fileA') p.joinpath('brokenLink').symlink_to('non-existing') p.joinpath('linkB').symlink_to('dirB') - p.joinpath('dirA', 'linkC').symlink_to(pathmod.join('..', 'dirB')) - p.joinpath('dirB', 'linkD').symlink_to(pathmod.join('..', 'dirB')) + p.joinpath('dirA', 'linkC').symlink_to(parser.join('..', 'dirB')) + p.joinpath('dirB', 'linkD').symlink_to(parser.join('..', 'dirB')) p.joinpath('brokenLinkLoop').symlink_to('brokenLinkLoop') def tearDown(self): @@ -1573,13 +1573,13 @@ def assertFileNotFound(self, func, *args, **kwargs): self.assertEqual(cm.exception.errno, errno.ENOENT) def assertEqualNormCase(self, path_a, path_b): - normcase = self.pathmod.normcase + normcase = self.parser.normcase self.assertEqual(normcase(path_a), normcase(path_b)) def test_samefile(self): - pathmod = self.pathmod - fileA_path = pathmod.join(self.base, 'fileA') - fileB_path = pathmod.join(self.base, 'dirB', 'fileB') + parser = self.parser + fileA_path = parser.join(self.base, 'fileA') + fileB_path = parser.join(self.base, 'dirB', 'fileB') p = self.cls(fileA_path) pp = self.cls(fileA_path) q = self.cls(fileB_path) @@ -1588,7 +1588,7 @@ def test_samefile(self): self.assertFalse(p.samefile(fileB_path)) self.assertFalse(p.samefile(q)) # Test the non-existent file case - non_existent = pathmod.join(self.base, 'foo') + non_existent = parser.join(self.base, 'foo') r = self.cls(non_existent) self.assertRaises(FileNotFoundError, p.samefile, r) self.assertRaises(FileNotFoundError, p.samefile, non_existent) @@ -2050,15 +2050,15 @@ def test_resolve_common(self): p.resolve(strict=True) self.assertEqual(cm.exception.errno, errno.ENOENT) # Non-strict - pathmod = self.pathmod + parser = self.parser self.assertEqualNormCase(str(p.resolve(strict=False)), - pathmod.join(self.base, 'foo')) + parser.join(self.base, 'foo')) p = P(self.base, 'foo', 'in', 'spam') self.assertEqualNormCase(str(p.resolve(strict=False)), - pathmod.join(self.base, 'foo', 'in', 'spam')) + parser.join(self.base, 'foo', 'in', 'spam')) p = P(self.base, '..', 'foo', 'in', 'spam') self.assertEqualNormCase(str(p.resolve(strict=False)), - pathmod.join(pathmod.dirname(self.base), 'foo', 'in', 'spam')) + parser.join(parser.dirname(self.base), 'foo', 'in', 'spam')) # These are all relative symlinks. p = P(self.base, 'dirB', 'fileB') self._check_resolve_relative(p, p) @@ -2073,7 +2073,7 @@ def test_resolve_common(self): self._check_resolve_relative(p, P(self.base, 'dirB', 'fileB', 'foo', 'in', 'spam'), False) p = P(self.base, 'dirA', 'linkC', '..', 'foo', 'in', 'spam') - if self.cls.pathmod is not posixpath: + if self.cls.parser is not posixpath: # In Windows, if linkY points to dirB, 'dirA\linkY\..' # resolves to 'dirA' without resolving linkY first. self._check_resolve_relative(p, P(self.base, 'dirA', 'foo', 'in', @@ -2085,7 +2085,7 @@ def test_resolve_common(self): # Now create absolute symlinks. d = self.tempdir() P(self.base, 'dirA', 'linkX').symlink_to(d) - P(self.base, str(d), 'linkY').symlink_to(self.pathmod.join(self.base, 'dirB')) + P(self.base, str(d), 'linkY').symlink_to(self.parser.join(self.base, 'dirB')) p = P(self.base, 'dirA', 'linkX', 'linkY', 'fileB') self._check_resolve_absolute(p, P(self.base, 'dirB', 'fileB')) # Non-strict @@ -2093,7 +2093,7 @@ def test_resolve_common(self): self._check_resolve_relative(p, P(self.base, 'dirB', 'foo', 'in', 'spam'), False) p = P(self.base, 'dirA', 'linkX', 'linkY', '..', 'foo', 'in', 'spam') - if self.cls.pathmod is not posixpath: + if self.cls.parser is not posixpath: # In Windows, if linkY points to dirB, 'dirA\linkY\..' # resolves to 'dirA' without resolving linkY first. self._check_resolve_relative(p, P(d, 'foo', 'in', 'spam'), False) @@ -2105,11 +2105,11 @@ def test_resolve_common(self): @needs_symlinks def test_resolve_dot(self): # See http://web.archive.org/web/20200623062557/https://bitbucket.org/pitrou/path… - pathmod = self.pathmod + parser = self.parser p = self.cls(self.base) p.joinpath('0').symlink_to('.', target_is_directory=True) - p.joinpath('1').symlink_to(pathmod.join('0', '0'), target_is_directory=True) - p.joinpath('2').symlink_to(pathmod.join('1', '1'), target_is_directory=True) + p.joinpath('1').symlink_to(parser.join('0', '0'), target_is_directory=True) + p.joinpath('2').symlink_to(parser.join('1', '1'), target_is_directory=True) q = p / '2' self.assertEqual(q.resolve(strict=True), p) r = q / '3' / '4' @@ -2137,11 +2137,11 @@ def test_resolve_loop(self): p = self.cls(self.base, 'linkZ', 'foo') self.assertEqual(p.resolve(strict=False), p) # Loops with absolute symlinks. - self.cls(self.base, 'linkU').symlink_to(self.pathmod.join(self.base, 'linkU/inside')) + self.cls(self.base, 'linkU').symlink_to(self.parser.join(self.base, 'linkU/inside')) self._check_symlink_loop(self.base, 'linkU') - self.cls(self.base, 'linkV').symlink_to(self.pathmod.join(self.base, 'linkV')) + self.cls(self.base, 'linkV').symlink_to(self.parser.join(self.base, 'linkV')) self._check_symlink_loop(self.base, 'linkV') - self.cls(self.base, 'linkW').symlink_to(self.pathmod.join(self.base, 'linkW/../linkW')) + self.cls(self.base, 'linkW').symlink_to(self.parser.join(self.base, 'linkW/../linkW')) self._check_symlink_loop(self.base, 'linkW') # Non-strict q = self.cls(self.base, 'linkW', 'foo') @@ -2313,11 +2313,11 @@ def test_is_char_device_false(self): def _check_complex_symlinks(self, link0_target): # Test solving a non-looping chain of symlinks (issue #19887). - pathmod = self.pathmod + parser = self.parser P = self.cls(self.base) - P.joinpath('link1').symlink_to(pathmod.join('link0', 'link0'), target_is_directory=True) - P.joinpath('link2').symlink_to(pathmod.join('link1', 'link1'), target_is_directory=True) - P.joinpath('link3').symlink_to(pathmod.join('link2', 'link2'), target_is_directory=True) + P.joinpath('link1').symlink_to(parser.join('link0', 'link0'), target_is_directory=True) + P.joinpath('link2').symlink_to(parser.join('link1', 'link1'), target_is_directory=True) + P.joinpath('link3').symlink_to(parser.join('link2', 'link2'), target_is_directory=True) P.joinpath('link0').symlink_to(link0_target, target_is_directory=True) # Resolve absolute paths. @@ -2367,7 +2367,7 @@ def test_complex_symlinks_relative(self): @needs_symlinks def test_complex_symlinks_relative_dot_dot(self): - self._check_complex_symlinks(self.pathmod.join('dirA', '..')) + self._check_complex_symlinks(self.parser.join('dirA', '..')) def setUpWalk(self): # Build:

1 0

gh-117303: Don't detach in `PyThreadState_DeleteCurrent()` (#117304)
by colesbury 29 Mar '24

29 Mar '24

https://github.com/python/cpython/commit/bfc57d43d8766120ba0c8f3f6d7b2ac681… commit: bfc57d43d8766120ba0c8f3f6d7b2ac681a81d8a branch: main author: Sam Gross <colesbury(a)gmail.com> committer: colesbury <colesbury(a)gmail.com> date: 2024-03-29T18:58:08-04:00 summary: gh-117303: Don't detach in `PyThreadState_DeleteCurrent()` (#117304) This fixes a crash in `test_threading.test_reinit_tls_after_fork()` when running with the GIL disabled. We already properly handle the case where the thread state is `_Py_THREAD_ATTACHED` in `tstate_delete_common()` -- we just need to remove an assertion. Keeping the thread attached means that a stop-the-world pause, such as for a `fork()`, won't commence until we remove our thread state from the interpreter's linked list. This prevents a crash when the child process tries to clean up the dead thread states. files: M Python/pystate.c diff --git a/Python/pystate.c b/Python/pystate.c index 8bec72779b2c24..925d1cff866f18 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1660,7 +1660,6 @@ static void tstate_delete_common(PyThreadState *tstate) { assert(tstate->_status.cleared && !tstate->_status.finalized); - assert(tstate->state != _Py_THREAD_ATTACHED); tstate_verify_not_active(tstate); assert(!_PyThreadState_IsRunningMain(tstate)); @@ -1740,7 +1739,6 @@ _PyThreadState_DeleteCurrent(PyThreadState *tstate) #ifdef Py_GIL_DISABLED _Py_qsbr_detach(((_PyThreadStateImpl *)tstate)->qsbr); #endif - tstate_set_detached(tstate, _Py_THREAD_DETACHED); current_fast_clear(tstate->interp->runtime); tstate_delete_common(tstate); _PyEval_ReleaseLock(tstate->interp, NULL);

1 0

gh-116664: In _warnings.c, make filters_version access thread-safe (#117374)
by erlend-aasland 29 Mar '24

29 Mar '24

https://github.com/python/cpython/commit/05e0b67a43c5c1778dc2643c8b7c12864e… commit: 05e0b67a43c5c1778dc2643c8b7c12864e135999 branch: main author: Erlend E. Aasland <erlend(a)python.org> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2024-03-29T21:23:28+01:00 summary: gh-116664: In _warnings.c, make filters_version access thread-safe (#117374) - assert that the lock is held in already_warned() - protect 'filters_version' increment in warnings_filters_mutated_impl() files: M Python/_warnings.c diff --git a/Python/_warnings.c b/Python/_warnings.c index ac3d3cc2d1246b..4c520252aa12a8 100644 --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -398,9 +398,9 @@ already_warned(PyInterpreterState *interp, PyObject *registry, PyObject *key, return -1; WarningsState *st = warnings_get_state(interp); - if (st == NULL) { - return -1; - } + assert(st != NULL); + _Py_CRITICAL_SECTION_ASSERT_MUTEX_LOCKED(&st->mutex); + PyObject *version_obj; if (PyDict_GetItemRef(registry, &_Py_ID(version), &version_obj) < 0) { return -1; @@ -1177,11 +1177,14 @@ warnings_filters_mutated_impl(PyObject *module) if (interp == NULL) { return NULL; } + WarningsState *st = warnings_get_state(interp); - if (st == NULL) { - return NULL; - } + assert(st != NULL); + + Py_BEGIN_CRITICAL_SECTION_MUT(&st->mutex); st->filters_version++; + Py_END_CRITICAL_SECTION(); + Py_RETURN_NONE; }

1 0

gh-117348: Refactored RawConfigParser._read for similicity and comprehensibility (#117372)
by jaraco 29 Mar '24

29 Mar '24

https://github.com/python/cpython/commit/019143fecbfc26e69800d28d2a9e3392a0… commit: 019143fecbfc26e69800d28d2a9e3392a051780b branch: main author: Jason R. Coombs <jaraco(a)jaraco.com> committer: jaraco <jaraco(a)jaraco.com> date: 2024-03-29T16:06:09-04:00 summary: gh-117348: Refactored RawConfigParser._read for similicity and comprehensibility (#117372) * Extract method for _read_inner, reducing complexity and indentation by 1. * Extract method for _raise_all and yield ParseErrors from _read_inner. Reduces complexity by 1 and reduces touch points for handling errors in _read_inner. * Prefer iterators to splat expansion and literal indexing. * Extract method for _strip_comments. Reduces complexity by 7. * Model the file lines in a class to encapsulate the comment status and cleaned value. * Encapsulate the read state as a dataclass * Extract _handle_continuation_line and _handle_rest methods. Reduces complexity by 8. * Reindent * At least for now, collect errors in the ReadState * Check for missing section header separately. * Extract methods for _handle_header and _handle_option. Reduces complexity by 6. * Remove unreachable code. Reduces complexity by 4. * Remove unreachable branch * Handle error condition early. Reduces complexity by 1. * Add blurb * Move _raise_all to ParsingError, as its behavior is most closely related to the exception class and not the reader. * Split _strip* into separate methods. * Refactor _strip_full to compute the strip just once and use 'not any' to determine the factor. * Replace use of 'sys.maxsize' with direct computation of the stripped value. * Extract has_comments as a dynamic property. * Implement clean as a cached property. * Model comment prefixes in the RawConfigParser within a prefixes namespace. * Use a regular expression to search for the first match. Avoids mutating variables and tricky logic and over-computing all of the starts when only the first is relevant. files: A Misc/NEWS.d/next/Library/2024-03-29-12-07-26.gh-issue-117348.WjCYvK.rst M Lib/configparser.py diff --git a/Lib/configparser.py b/Lib/configparser.py index 3040e1fbe5b9c1..d0326c60e9b907 100644 --- a/Lib/configparser.py +++ b/Lib/configparser.py @@ -145,12 +145,15 @@ from collections.abc import MutableMapping from collections import ChainMap as _ChainMap +import contextlib +from dataclasses import dataclass, field import functools import io import itertools import os import re import sys +from typing import Iterable __all__ = ("NoSectionError", "DuplicateOptionError", "DuplicateSectionError", "NoOptionError", "InterpolationError", "InterpolationDepthError", @@ -302,15 +305,33 @@ def __init__(self, option, section, rawval): class ParsingError(Error): """Raised when a configuration file does not follow legal syntax.""" - def __init__(self, source): + def __init__(self, source, *args): super().__init__(f'Source contains parsing errors: {source!r}') self.source = source self.errors = [] self.args = (source, ) + if args: + self.append(*args) def append(self, lineno, line): self.errors.append((lineno, line)) - self.message += '\n\t[line %2d]: %s' % (lineno, line) + self.message += '\n\t[line %2d]: %s' % (lineno, repr(line)) + + def combine(self, others): + for other in others: + for error in other.errors: + self.append(*error) + return self + + @staticmethod + def _raise_all(exceptions: Iterable['ParsingError']): + """ + Combine any number of ParsingErrors into one and raise it. + """ + exceptions = iter(exceptions) + with contextlib.suppress(StopIteration): + raise next(exceptions).combine(exceptions) + class MissingSectionHeaderError(ParsingError): @@ -517,6 +538,55 @@ def _interpolate_some(self, parser, option, accum, rest, section, map, "found: %r" % (rest,)) +@dataclass +class _ReadState: + elements_added : set[str] = field(default_factory=set) + cursect : dict[str, str] | None = None + sectname : str | None = None + optname : str | None = None + lineno : int = 0 + indent_level : int = 0 + errors : list[ParsingError] = field(default_factory=list) + + +@dataclass +class _Prefixes: + full : Iterable[str] + inline : Iterable[str] + + +class _Line(str): + + def __new__(cls, val, *args, **kwargs): + return super().__new__(cls, val) + + def __init__(self, val, prefixes: _Prefixes): + self.prefixes = prefixes + + @functools.cached_property + def clean(self): + return self._strip_full() and self._strip_inline() + + @property + def has_comments(self): + return self.strip() != self.clean + + def _strip_inline(self): + """ + Search for the earliest prefix at the beginning of the line or following a space. + """ + matcher = re.compile( + '|'.join(fr'(^|\s)({re.escape(prefix)})' for prefix in self.prefixes.inline) + # match nothing if no prefixes + or '(?!)' + ) + match = matcher.search(self) + return self[:match.start() if match else None].strip() + + def _strip_full(self): + return '' if any(map(self.strip().startswith, self.prefixes.full)) else True + + class RawConfigParser(MutableMapping): """ConfigParser that does not do interpolation.""" @@ -583,8 +653,10 @@ def __init__(self, defaults=None, dict_type=_default_dict, else: self._optcre = re.compile(self._OPT_TMPL.format(delim=d), re.VERBOSE) - self._comment_prefixes = tuple(comment_prefixes or ()) - self._inline_comment_prefixes = tuple(inline_comment_prefixes or ()) + self._prefixes = _Prefixes( + full=tuple(comment_prefixes or ()), + inline=tuple(inline_comment_prefixes or ()), + ) self._strict = strict self._allow_no_value = allow_no_value self._empty_lines_in_values = empty_lines_in_values @@ -975,147 +1047,117 @@ def _read(self, fp, fpname): in an otherwise empty line or may be entered in lines holding values or section names. Please note that comments get stripped off when reading configuration files. """ - elements_added = set() - cursect = None # None, or a dictionary - sectname = None - optname = None - lineno = 0 - indent_level = 0 - e = None # None, or an exception try: - for lineno, line in enumerate(fp, start=1): - comment_start = sys.maxsize - # strip inline comments - inline_prefixes = {p: -1 for p in self._inline_comment_prefixes} - while comment_start == sys.maxsize and inline_prefixes: - next_prefixes = {} - for prefix, index in inline_prefixes.items(): - index = line.find(prefix, index+1) - if index == -1: - continue - next_prefixes[prefix] = index - if index == 0 or (index > 0 and line[index-1].isspace()): - comment_start = min(comment_start, index) - inline_prefixes = next_prefixes - # strip full line comments - for prefix in self._comment_prefixes: - if line.strip().startswith(prefix): - comment_start = 0 - break - if comment_start == sys.maxsize: - comment_start = None - value = line[:comment_start].strip() - if not value: - if self._empty_lines_in_values: - # add empty line to the value, but only if there was no - # comment on the line - if (comment_start is None and - cursect is not None and - optname and - cursect[optname] is not None): - cursect[optname].append('') # newlines added at join - else: - # empty line marks end of value - indent_level = sys.maxsize - continue - # continuation line? - first_nonspace = self.NONSPACECRE.search(line) - cur_indent_level = first_nonspace.start() if first_nonspace else 0 - if (cursect is not None and optname and - cur_indent_level > indent_level): - if cursect[optname] is None: - raise MultilineContinuationError(fpname, lineno, line) - cursect[optname].append(value) - # a section header or option header? - else: - if self._allow_unnamed_section and cursect is None: - sectname = UNNAMED_SECTION - cursect = self._dict() - self._sections[sectname] = cursect - self._proxies[sectname] = SectionProxy(self, sectname) - elements_added.add(sectname) - - indent_level = cur_indent_level - # is it a section header? - mo = self.SECTCRE.match(value) - if mo: - sectname = mo.group('header') - if sectname in self._sections: - if self._strict and sectname in elements_added: - raise DuplicateSectionError(sectname, fpname, - lineno) - cursect = self._sections[sectname] - elements_added.add(sectname) - elif sectname == self.default_section: - cursect = self._defaults - else: - cursect = self._dict() - self._sections[sectname] = cursect - self._proxies[sectname] = SectionProxy(self, sectname) - elements_added.add(sectname) - # So sections can't start with a continuation line - optname = None - # no section header? - elif cursect is None: - raise MissingSectionHeaderError(fpname, lineno, line) - # an option line? - else: - indent_level = cur_indent_level - # is it a section header? - mo = self.SECTCRE.match(value) - if mo: - sectname = mo.group('header') - if sectname in self._sections: - if self._strict and sectname in elements_added: - raise DuplicateSectionError(sectname, fpname, - lineno) - cursect = self._sections[sectname] - elements_added.add(sectname) - elif sectname == self.default_section: - cursect = self._defaults - else: - cursect = self._dict() - self._sections[sectname] = cursect - self._proxies[sectname] = SectionProxy(self, sectname) - elements_added.add(sectname) - # So sections can't start with a continuation line - optname = None - # no section header in the file? - elif cursect is None: - raise MissingSectionHeaderError(fpname, lineno, line) - # an option line? - else: - mo = self._optcre.match(value) - if mo: - optname, vi, optval = mo.group('option', 'vi', 'value') - if not optname: - e = self._handle_error(e, fpname, lineno, line) - optname = self.optionxform(optname.rstrip()) - if (self._strict and - (sectname, optname) in elements_added): - raise DuplicateOptionError(sectname, optname, - fpname, lineno) - elements_added.add((sectname, optname)) - # This check is fine because the OPTCRE cannot - # match if it would set optval to None - if optval is not None: - optval = optval.strip() - cursect[optname] = [optval] - else: - # valueless option handling - cursect[optname] = None - else: - # a non-fatal parsing error occurred. set up the - # exception but keep going. the exception will be - # raised at the end of the file and will contain a - # list of all bogus lines - e = self._handle_error(e, fpname, lineno, line) + ParsingError._raise_all(self._read_inner(fp, fpname)) finally: self._join_multiline_values() - # if any parsing errors occurred, raise an exception - if e: - raise e + + def _read_inner(self, fp, fpname): + st = _ReadState() + + Line = functools.partial(_Line, prefixes=self._prefixes) + for st.lineno, line in enumerate(map(Line, fp), start=1): + if not line.clean: + if self._empty_lines_in_values: + # add empty line to the value, but only if there was no + # comment on the line + if (not line.has_comments and + st.cursect is not None and + st.optname and + st.cursect[st.optname] is not None): + st.cursect[st.optname].append('') # newlines added at join + else: + # empty line marks end of value + st.indent_level = sys.maxsize + continue + + first_nonspace = self.NONSPACECRE.search(line) + st.cur_indent_level = first_nonspace.start() if first_nonspace else 0 + + if self._handle_continuation_line(st, line, fpname): + continue + + self._handle_rest(st, line, fpname) + + return st.errors + + def _handle_continuation_line(self, st, line, fpname): + # continuation line? + is_continue = (st.cursect is not None and st.optname and + st.cur_indent_level > st.indent_level) + if is_continue: + if st.cursect[st.optname] is None: + raise MultilineContinuationError(fpname, st.lineno, line) + st.cursect[st.optname].append(line.clean) + return is_continue + + def _handle_rest(self, st, line, fpname): + # a section header or option header? + if self._allow_unnamed_section and st.cursect is None: + st.sectname = UNNAMED_SECTION + st.cursect = self._dict() + self._sections[st.sectname] = st.cursect + self._proxies[st.sectname] = SectionProxy(self, st.sectname) + st.elements_added.add(st.sectname) + + st.indent_level = st.cur_indent_level + # is it a section header? + mo = self.SECTCRE.match(line.clean) + + if not mo and st.cursect is None: + raise MissingSectionHeaderError(fpname, st.lineno, line) + + self._handle_header(st, mo, fpname) if mo else self._handle_option(st, line, fpname) + + def _handle_header(self, st, mo, fpname): + st.sectname = mo.group('header') + if st.sectname in self._sections: + if self._strict and st.sectname in st.elements_added: + raise DuplicateSectionError(st.sectname, fpname, + st.lineno) + st.cursect = self._sections[st.sectname] + st.elements_added.add(st.sectname) + elif st.sectname == self.default_section: + st.cursect = self._defaults + else: + st.cursect = self._dict() + self._sections[st.sectname] = st.cursect + self._proxies[st.sectname] = SectionProxy(self, st.sectname) + st.elements_added.add(st.sectname) + # So sections can't start with a continuation line + st.optname = None + + def _handle_option(self, st, line, fpname): + # an option line? + st.indent_level = st.cur_indent_level + + mo = self._optcre.match(line.clean) + if not mo: + # a non-fatal parsing error occurred. set up the + # exception but keep going. the exception will be + # raised at the end of the file and will contain a + # list of all bogus lines + st.errors.append(ParsingError(fpname, st.lineno, line)) + return + + st.optname, vi, optval = mo.group('option', 'vi', 'value') + if not st.optname: + st.errors.append(ParsingError(fpname, st.lineno, line)) + st.optname = self.optionxform(st.optname.rstrip()) + if (self._strict and + (st.sectname, st.optname) in st.elements_added): + raise DuplicateOptionError(st.sectname, st.optname, + fpname, st.lineno) + st.elements_added.add((st.sectname, st.optname)) + # This check is fine because the OPTCRE cannot + # match if it would set optval to None + if optval is not None: + optval = optval.strip() + st.cursect[st.optname] = [optval] + else: + # valueless option handling + st.cursect[st.optname] = None def _join_multiline_values(self): defaults = self.default_section, self._defaults @@ -1135,12 +1177,6 @@ def _read_defaults(self, defaults): for key, value in defaults.items(): self._defaults[self.optionxform(key)] = value - def _handle_error(self, exc, fpname, lineno, line): - if not exc: - exc = ParsingError(fpname) - exc.append(lineno, repr(line)) - return exc - def _unify_values(self, section, vars): """Create a sequence of lookups with 'vars' taking priority over the 'section' which takes priority over the DEFAULTSECT. diff --git a/Misc/NEWS.d/next/Library/2024-03-29-12-07-26.gh-issue-117348.WjCYvK.rst b/Misc/NEWS.d/next/Library/2024-03-29-12-07-26.gh-issue-117348.WjCYvK.rst new file mode 100644 index 00000000000000..cd3006c3b7b8f0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-03-29-12-07-26.gh-issue-117348.WjCYvK.rst @@ -0,0 +1,2 @@ +Refactored :meth:`configparser.RawConfigParser._read` to reduce cyclometric +complexity and improve comprehensibility.

1 0

gh-117300: Use stop the world to make `sys._current_frames` and `sys._current_exceptions` thread-safe. (#117301)
by colesbury 29 Mar '24

29 Mar '24

https://github.com/python/cpython/commit/01bd74eadbc4ff839d39762fae6366f50c… commit: 01bd74eadbc4ff839d39762fae6366f50c1e116e branch: main author: Sam Gross <colesbury(a)gmail.com> committer: colesbury <colesbury(a)gmail.com> date: 2024-03-29T15:33:06-04:00 summary: gh-117300: Use stop the world to make `sys._current_frames` and `sys._current_exceptions` thread-safe. (#117301) This adds a stop the world pause to make the two functions thread-safe when the GIL is disabled in the free-threaded build. Additionally, the main test thread may call `sys._current_exceptions()` as soon as `g_raised.set()` is called. The background thread may not yet reach the `leave_g.wait()` line. files: M Lib/test/test_sys.py M Python/pystate.c diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 37c16cd1047885..f6f23b0afc34c6 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -562,7 +562,8 @@ def g456(): # And the next record must be for g456(). filename, lineno, funcname, sourceline = stack[i+1] self.assertEqual(funcname, "g456") - self.assertTrue(sourceline.startswith("if leave_g.wait(")) + self.assertTrue((sourceline.startswith("if leave_g.wait(") or + sourceline.startswith("g_raised.set()"))) finally: # Reap the spawned thread. leave_g.set() diff --git a/Python/pystate.c b/Python/pystate.c index 8489f53c6e3e34..8bec72779b2c24 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -2408,6 +2408,7 @@ _PyThread_CurrentFrames(void) * Because these lists can mutate even when the GIL is held, we * need to grab head_mutex for the duration. */ + _PyEval_StopTheWorldAll(runtime); HEAD_LOCK(runtime); PyInterpreterState *i; for (i = runtime->interpreters.head; i != NULL; i = i->next) { @@ -2441,6 +2442,7 @@ _PyThread_CurrentFrames(void) done: HEAD_UNLOCK(runtime); + _PyEval_StartTheWorldAll(runtime); return result; } @@ -2472,6 +2474,7 @@ _PyThread_CurrentExceptions(void) * Because these lists can mutate even when the GIL is held, we * need to grab head_mutex for the duration. */ + _PyEval_StopTheWorldAll(runtime); HEAD_LOCK(runtime); PyInterpreterState *i; for (i = runtime->interpreters.head; i != NULL; i = i->next) { @@ -2504,6 +2507,7 @@ _PyThread_CurrentExceptions(void) done: HEAD_UNLOCK(runtime); + _PyEval_StartTheWorldAll(runtime); return result; }

1 0