From webhook-mailer at python.org Sat Aug 1 04:18:51 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Sat, 01 Aug 2020 08:18:51 -0000 Subject: [Python-checkins] bpo-41421: Algebraic simplification for random.paretovariate() (GH-21695) Message-ID: https://github.com/python/cpython/commit/5c3270939c09e4c8e80fd26449b718a998701912 commit: 5c3270939c09e4c8e80fd26449b718a998701912 branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-01T01:18:26-07:00 summary: bpo-41421: Algebraic simplification for random.paretovariate() (GH-21695) files: A Misc/NEWS.d/next/Library/2020-08-01-00-51-15.bpo-41421.dHKRVB.rst M Lib/random.py diff --git a/Lib/random.py b/Lib/random.py index a6454f520df0a..37f71110403ad 100644 --- a/Lib/random.py +++ b/Lib/random.py @@ -749,7 +749,7 @@ def paretovariate(self, alpha): # Jain, pg. 495 u = 1.0 - self.random() - return 1.0 / u ** (1.0 / alpha) + return u ** (-1.0 / alpha) def weibullvariate(self, alpha, beta): """Weibull distribution. diff --git a/Misc/NEWS.d/next/Library/2020-08-01-00-51-15.bpo-41421.dHKRVB.rst b/Misc/NEWS.d/next/Library/2020-08-01-00-51-15.bpo-41421.dHKRVB.rst new file mode 100644 index 0000000000000..cf291c60d8ad5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-01-00-51-15.bpo-41421.dHKRVB.rst @@ -0,0 +1,3 @@ +Make an algebraic simplification to random.paretovariate(). It now is +slightly less subject to round-off error and is slightly faster. Inputs that +used to cause ZeroDivisionError now cause an OverflowError instead. From webhook-mailer at python.org Sun Aug 2 15:04:00 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Sun, 02 Aug 2020 19:04:00 -0000 Subject: [Python-checkins] random module: Convert a "while 1" to "while True (GH-21700) Message-ID: https://github.com/python/cpython/commit/6a613f90bf13038255bca028feeab6fad816edfd commit: 6a613f90bf13038255bca028feeab6fad816edfd branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-02T12:03:32-07:00 summary: random module: Convert a "while 1" to "while True (GH-21700) files: M Lib/random.py diff --git a/Lib/random.py b/Lib/random.py index 37f71110403ad..3ea369b81b3e5 100644 --- a/Lib/random.py +++ b/Lib/random.py @@ -682,7 +682,7 @@ def gammavariate(self, alpha, beta): bbb = alpha - LOG4 ccc = alpha + ainv - while 1: + while True: u1 = random() if not 1e-7 < u1 < 0.9999999: continue From webhook-mailer at python.org Sun Aug 2 18:33:03 2020 From: webhook-mailer at python.org (Luciano Ramalho) Date: Sun, 02 Aug 2020 22:33:03 -0000 Subject: [Python-checkins] bpo-40979: refactored typing.rst; (mostly) same content, new sub-sections and ordering (#21574) Message-ID: https://github.com/python/cpython/commit/ab72fdeb82c3ab045b480cd4bb4f928c12653ecb commit: ab72fdeb82c3ab045b480cd4bb4f928c12653ecb branch: master author: Luciano Ramalho committer: GitHub date: 2020-08-02T15:32:36-07:00 summary: bpo-40979: refactored typing.rst; (mostly) same content, new sub-sections and ordering (#21574) Also added PEP 585 deprecation notes. files: A Misc/NEWS.d/next/Documentation/2020-07-21-15-23-30.bpo-40979.pLA8rO.rst M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index e5143c5f8d9e4..44b537f1669e1 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1,3 +1,4 @@ +======================================== :mod:`typing` --- Support for type hints ======================================== @@ -34,7 +35,7 @@ In the function ``greeting``, the argument ``name`` is expected to be of type arguments. Type aliases ------------- +============ A type alias is defined by assigning the type to the alias. In this example, ``Vector`` and ``List[float]`` will be treated as interchangeable synonyms:: @@ -72,7 +73,7 @@ Note that ``None`` as a type hint is a special case and is replaced by .. _distinct: NewType -------- +======= Use the :func:`NewType` helper function to create distinct types:: @@ -149,7 +150,7 @@ See :pep:`484` for more details. .. versionadded:: 3.5.2 Callable --------- +======== Frameworks expecting callback functions of specific signatures might be type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. @@ -172,7 +173,7 @@ for the list of arguments in the type hint: ``Callable[..., ReturnType]``. .. _generics: Generics --------- +======== Since type information about objects kept in containers cannot be statically inferred in a generic way, abstract base classes have been extended to support @@ -199,7 +200,7 @@ called :class:`TypeVar`. User-defined generic types --------------------------- +========================== A user-defined class can be defined as a generic class. @@ -317,7 +318,7 @@ comparable for equality. The :data:`Any` type --------------------- +==================== A special kind of type is :data:`Any`. A static type checker will treat every type as being compatible with :data:`Any` and :data:`Any` as being @@ -395,7 +396,7 @@ manner. Use :data:`Any` to indicate that a value is dynamically typed. Nominal vs structural subtyping -------------------------------- +=============================== Initially :pep:`484` defined Python static type system as using *nominal subtyping*. This means that a class ``A`` is allowed where @@ -434,106 +435,152 @@ Moreover, by subclassing a special class :class:`Protocol`, a user can define new custom protocols to fully enjoy structural subtyping (see examples below). +Module contents +=============== -Classes, functions, and decorators ----------------------------------- +The module defines the following classes, functions and decorators. -The module defines the following classes, functions and decorators: +.. note:: -.. class:: TypeVar + This module defines several types that are subclasses of pre-existing + standard library classes which also extend :class:`Generic` + to support type variables inside ``[]``. + These types became redundant in Python 3.9 when the + corresponding pre-existing classes were enhanced to support ``[]``. - Type variable. + The redundant types are deprecated as of Python 3.9 but no + deprecation warnings will be issued by the interpreter. + It is expected that type checkers will flag the deprecated types + when the checked program targets Python 3.9 or newer. - Usage:: + The deprecated types will be removed from the :mod:`typing` module + in the first Python version released 5 years after the release of Python 3.9.0. + See details in :pep:`585`?*Type Hinting Generics In Standard Collections*. - T = TypeVar('T') # Can be anything - A = TypeVar('A', str, bytes) # Must be str or bytes - Type variables exist primarily for the benefit of static type - checkers. They serve as the parameters for generic types as well - as for generic function definitions. See class Generic for more - information on generic types. Generic functions work as follows:: +Special typing primitives +------------------------- - def repeat(x: T, n: int) -> Sequence[T]: - """Return a list containing n references to x.""" - return [x]*n +Special types +""""""""""""" - def longest(x: A, y: A) -> A: - """Return the longest of two strings.""" - return x if len(x) >= len(y) else y +These can be used as types in annotations and do not support ``[]``. - The latter example's signature is essentially the overloading - of ``(str, str) -> str`` and ``(bytes, bytes) -> bytes``. Also note - that if the arguments are instances of some subclass of :class:`str`, - the return type is still plain :class:`str`. +.. data:: Any - At runtime, ``isinstance(x, T)`` will raise :exc:`TypeError`. In general, - :func:`isinstance` and :func:`issubclass` should not be used with types. + Special type indicating an unconstrained type. - Type variables may be marked covariant or contravariant by passing - ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more - details. By default type variables are invariant. Alternatively, - a type variable may specify an upper bound using ``bound=``. - This means that an actual type substituted (explicitly or implicitly) - for the type variable must be a subclass of the boundary type, - see :pep:`484`. + * Every type is compatible with :data:`Any`. + * :data:`Any` is compatible with every type. -.. class:: Generic +.. data:: NoReturn - Abstract base class for generic types. + Special type indicating that a function never returns. + For example:: - A generic type is typically declared by inheriting from an - instantiation of this class with one or more type variables. - For example, a generic mapping type might be defined as:: + from typing import NoReturn - class Mapping(Generic[KT, VT]): - def __getitem__(self, key: KT) -> VT: - ... - # Etc. + def stop() -> NoReturn: + raise RuntimeError('no way') - This class can then be used as follows:: + .. versionadded:: 3.5.4 + .. versionadded:: 3.6.2 - X = TypeVar('X') - Y = TypeVar('Y') +Special forms +""""""""""""" - def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: - try: - return mapping[key] - except KeyError: - return default +These can be used as types in annotations using ``[]``, each having a unique syntax. -.. class:: Protocol(Generic) +.. data:: Tuple - Base class for protocol classes. Protocol classes are defined like this:: + Tuple type; ``Tuple[X, Y]`` is the type of a tuple of two items + with the first item of type X and the second of type Y. The type of + the empty tuple can be written as ``Tuple[()]``. - class Proto(Protocol): - def meth(self) -> int: - ... + Example: ``Tuple[T1, T2]`` is a tuple of two elements corresponding + to type variables T1 and T2. ``Tuple[int, float, str]`` is a tuple + of an int, a float and a string. - Such classes are primarily used with static type checkers that recognize - structural subtyping (static duck-typing), for example:: + To specify a variable-length tuple of homogeneous type, + use literal ellipsis, e.g. ``Tuple[int, ...]``. A plain :data:`Tuple` + is equivalent to ``Tuple[Any, ...]``, and in turn to :class:`tuple`. - class C: - def meth(self) -> int: - return 0 + .. deprecated:: 3.9 + :class:`builtins.tuple ` now supports ``[]``. See :pep:`585`. - def func(x: Proto) -> int: - return x.meth() +.. data:: Union - func(C()) # Passes static type check + Union type; ``Union[X, Y]`` means either X or Y. - See :pep:`544` for details. Protocol classes decorated with - :func:`runtime_checkable` (described later) act as simple-minded runtime - protocols that check only the presence of given attributes, ignoring their - type signatures. + To define a union, use e.g. ``Union[int, str]``. Details: - Protocol classes can be generic, for example:: + * The arguments must be types and there must be at least one. - class GenProto(Protocol[T]): - def meth(self) -> T: - ... + * Unions of unions are flattened, e.g.:: - .. versionadded:: 3.8 + Union[Union[int, str], float] == Union[int, str, float] + + * Unions of a single argument vanish, e.g.:: + + Union[int] == int # The constructor actually returns int + + * Redundant arguments are skipped, e.g.:: + + Union[int, str, int] == Union[int, str] + + * When comparing unions, the argument order is ignored, e.g.:: + + Union[int, str] == Union[str, int] + + * You cannot subclass or instantiate a union. + + * You cannot write ``Union[X][Y]``. + + * You can use ``Optional[X]`` as a shorthand for ``Union[X, None]``. + + .. versionchanged:: 3.7 + Don't remove explicit subclasses from unions at runtime. + +.. data:: Optional + + Optional type. + + ``Optional[X]`` is equivalent to ``Union[X, None]``. + + Note that this is not the same concept as an optional argument, + which is one that has a default. An optional argument with a + default does not require the ``Optional`` qualifier on its type + annotation just because it is optional. For example:: + + def foo(arg: int = 0) -> None: + ... + + On the other hand, if an explicit value of ``None`` is allowed, the + use of ``Optional`` is appropriate, whether the argument is optional + or not. For example:: + + def foo(arg: Optional[int] = None) -> None: + ... + +.. data:: Callable + + Callable type; ``Callable[[int], str]`` is a function of (int) -> str. + + The subscription syntax must always be used with exactly two + values: the argument list and the return type. The argument list + must be a list of types or an ellipsis; the return type must be + a single type. + + There is no syntax to indicate optional or keyword arguments; + such function types are rarely used as callback types. + ``Callable[..., ReturnType]`` (literal ellipsis) can be used to + type hint a callable taking any number of arguments and returning + ``ReturnType``. A plain :data:`Callable` is equivalent to + ``Callable[..., Any]``, and in turn to + :class:`collections.abc.Callable`. + + .. deprecated:: 3.9 + :class:`collections.abc.Callable` now supports ``[]``. See :pep:`585`. .. class:: Type(Generic[CT_co]) @@ -577,374 +624,342 @@ The module defines the following classes, functions and decorators: .. versionadded:: 3.5.2 -.. class:: Iterable(Generic[T_co]) + .. deprecated:: 3.9 + :class:`builtins.type ` now supports ``[]``. See :pep:`585`. - A generic version of :class:`collections.abc.Iterable`. +.. data:: Literal -.. class:: Iterator(Iterable[T_co]) + A type that can be used to indicate to type checkers that the + corresponding variable or function parameter has a value equivalent to + the provided literal (or one of several literals). For example:: - A generic version of :class:`collections.abc.Iterator`. + def validate_simple(data: Any) -> Literal[True]: # always returns True + ... -.. class:: Reversible(Iterable[T_co]) + MODE = Literal['r', 'rb', 'w', 'wb'] + def open_helper(file: str, mode: MODE) -> str: + ... - A generic version of :class:`collections.abc.Reversible`. + open_helper('/some/path', 'r') # Passes type check + open_helper('/other/path', 'typo') # Error in type checker -.. class:: SupportsInt + ``Literal[...]`` cannot be subclassed. At runtime, an arbitrary value + is allowed as type argument to ``Literal[...]``, but type checkers may + impose restrictions. See :pep:`586` for more details about literal types. - An ABC with one abstract method ``__int__``. + .. versionadded:: 3.8 -.. class:: SupportsFloat +.. data:: ClassVar - An ABC with one abstract method ``__float__``. + Special type construct to mark class variables. -.. class:: SupportsComplex + As introduced in :pep:`526`, a variable annotation wrapped in ClassVar + indicates that a given attribute is intended to be used as a class variable + and should not be set on instances of that class. Usage:: - An ABC with one abstract method ``__complex__``. + class Starship: + stats: ClassVar[Dict[str, int]] = {} # class variable + damage: int = 10 # instance variable -.. class:: SupportsBytes + :data:`ClassVar` accepts only types and cannot be further subscribed. - An ABC with one abstract method ``__bytes__``. + :data:`ClassVar` is not a class itself, and should not + be used with :func:`isinstance` or :func:`issubclass`. + :data:`ClassVar` does not change Python runtime behavior, but + it can be used by third-party type checkers. For example, a type checker + might flag the following code as an error:: -.. class:: SupportsIndex + enterprise_d = Starship(3000) + enterprise_d.stats = {} # Error, setting class variable on instance + Starship.stats = {} # This is OK - An ABC with one abstract method ``__index__``. + .. versionadded:: 3.5.3 - .. versionadded:: 3.8 +.. data:: Final -.. class:: SupportsAbs + A special typing construct to indicate to type checkers that a name + cannot be re-assigned or overridden in a subclass. For example:: - An ABC with one abstract method ``__abs__`` that is covariant - in its return type. + MAX_SIZE: Final = 9000 + MAX_SIZE += 1 # Error reported by type checker -.. class:: SupportsRound + class Connection: + TIMEOUT: Final[int] = 10 - An ABC with one abstract method ``__round__`` - that is covariant in its return type. + class FastConnector(Connection): + TIMEOUT = 1 # Error reported by type checker -.. class:: Container(Generic[T_co]) + There is no runtime checking of these properties. See :pep:`591` for + more details. - A generic version of :class:`collections.abc.Container`. + .. versionadded:: 3.8 -.. class:: Hashable +.. data:: Annotated - An alias to :class:`collections.abc.Hashable` + A type, introduced in :pep:`593` (``Flexible function and variable + annotations``), to decorate existing types with context-specific metadata + (possibly multiple pieces of it, as ``Annotated`` is variadic). + Specifically, a type ``T`` can be annotated with metadata ``x`` via the + typehint ``Annotated[T, x]``. This metadata can be used for either static + analysis or at runtime. If a library (or tool) encounters a typehint + ``Annotated[T, x]`` and has no special logic for metadata ``x``, it + should ignore it and simply treat the type as ``T``. Unlike the + ``no_type_check`` functionality that currently exists in the ``typing`` + module which completely disables typechecking annotations on a function + or a class, the ``Annotated`` type allows for both static typechecking + of ``T`` (e.g., via mypy or Pyre, which can safely ignore ``x``) + together with runtime access to ``x`` within a specific application. -.. class:: Sized + Ultimately, the responsibility of how to interpret the annotations (if + at all) is the responsibility of the tool or library encountering the + ``Annotated`` type. A tool or library encountering an ``Annotated`` type + can scan through the annotations to determine if they are of interest + (e.g., using ``isinstance()``). - An alias to :class:`collections.abc.Sized` + When a tool or a library does not support annotations or encounters an + unknown annotation it should just ignore it and treat annotated type as + the underlying type. -.. class:: Collection(Sized, Iterable[T_co], Container[T_co]) + It's up to the tool consuming the annotations to decide whether the + client is allowed to have several annotations on one type and how to + merge those annotations. - A generic version of :class:`collections.abc.Collection` + Since the ``Annotated`` type allows you to put several annotations of + the same (or different) type(s) on any node, the tools or libraries + consuming those annotations are in charge of dealing with potential + duplicates. For example, if you are doing value range analysis you might + allow this:: - .. versionadded:: 3.6.0 + T1 = Annotated[int, ValueRange(-10, 5)] + T2 = Annotated[T1, ValueRange(-20, 3)] -.. class:: AbstractSet(Sized, Collection[T_co]) + Passing ``include_extras=True`` to :func:`get_type_hints` lets one + access the extra annotations at runtime. - A generic version of :class:`collections.abc.Set`. + The details of the syntax: -.. class:: MutableSet(AbstractSet[T]) + * The first argument to ``Annotated`` must be a valid type - A generic version of :class:`collections.abc.MutableSet`. + * Multiple type annotations are supported (``Annotated`` supports variadic + arguments):: -.. class:: Mapping(Sized, Collection[KT], Generic[VT_co]) + Annotated[int, ValueRange(3, 10), ctype("char")] - A generic version of :class:`collections.abc.Mapping`. - This type can be used as follows:: + * ``Annotated`` must be called with at least two arguments ( + ``Annotated[int]`` is not valid) - def get_position_in_index(word_list: Mapping[str, int], word: str) -> int: - return word_list[word] + * The order of the annotations is preserved and matters for equality + checks:: -.. class:: MutableMapping(Mapping[KT, VT]) + Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[ + int, ctype("char"), ValueRange(3, 10) + ] - A generic version of :class:`collections.abc.MutableMapping`. + * Nested ``Annotated`` types are flattened, with metadata ordered + starting with the innermost annotation:: -.. class:: Sequence(Reversible[T_co], Collection[T_co]) + Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[ + int, ValueRange(3, 10), ctype("char") + ] - A generic version of :class:`collections.abc.Sequence`. + * Duplicated annotations are not removed:: -.. class:: MutableSequence(Sequence[T]) + Annotated[int, ValueRange(3, 10)] != Annotated[ + int, ValueRange(3, 10), ValueRange(3, 10) + ] - A generic version of :class:`collections.abc.MutableSequence`. + * ``Annotated`` can be used with nested and generic aliases:: -.. class:: ByteString(Sequence[int]) + T = TypeVar('T') + Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] + V = Vec[int] - A generic version of :class:`collections.abc.ByteString`. + V == Annotated[List[Tuple[int, int]], MaxLen(10)] - This type represents the types :class:`bytes`, :class:`bytearray`, - and :class:`memoryview` of byte sequences. + .. versionadded:: 3.9 - As a shorthand for this type, :class:`bytes` can be used to - annotate arguments of any of the types mentioned above. +Building generic types +"""""""""""""""""""""" -.. class:: Deque(deque, MutableSequence[T]) +These are not used in annotations. They are building blocks for creating generic types. - A generic version of :class:`collections.deque`. +.. class:: Generic - .. versionadded:: 3.5.4 - .. versionadded:: 3.6.1 + Abstract base class for generic types. -.. class:: List(list, MutableSequence[T]) + A generic type is typically declared by inheriting from an + instantiation of this class with one or more type variables. + For example, a generic mapping type might be defined as:: - Generic version of :class:`list`. - Useful for annotating return types. To annotate arguments it is preferred - to use an abstract collection type such as :class:`Sequence` or - :class:`Iterable`. + class Mapping(Generic[KT, VT]): + def __getitem__(self, key: KT) -> VT: + ... + # Etc. - This type may be used as follows:: + This class can then be used as follows:: - T = TypeVar('T', int, float) + X = TypeVar('X') + Y = TypeVar('Y') - def vec2(x: T, y: T) -> List[T]: - return [x, y] + def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: + try: + return mapping[key] + except KeyError: + return default - def keep_positives(vector: Sequence[T]) -> List[T]: - return [item for item in vector if item > 0] +.. class:: TypeVar -.. class:: Set(set, MutableSet[T]) + Type variable. - A generic version of :class:`builtins.set `. - Useful for annotating return types. To annotate arguments it is preferred - to use an abstract collection type such as :class:`AbstractSet`. + Usage:: -.. class:: FrozenSet(frozenset, AbstractSet[T_co]) + T = TypeVar('T') # Can be anything + A = TypeVar('A', str, bytes) # Must be str or bytes - A generic version of :class:`builtins.frozenset `. + Type variables exist primarily for the benefit of static type + checkers. They serve as the parameters for generic types as well + as for generic function definitions. See class Generic for more + information on generic types. Generic functions work as follows:: -.. class:: MappingView(Sized, Iterable[T_co]) + def repeat(x: T, n: int) -> Sequence[T]: + """Return a list containing n references to x.""" + return [x]*n - A generic version of :class:`collections.abc.MappingView`. + def longest(x: A, y: A) -> A: + """Return the longest of two strings.""" + return x if len(x) >= len(y) else y -.. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co]) + The latter example's signature is essentially the overloading + of ``(str, str) -> str`` and ``(bytes, bytes) -> bytes``. Also note + that if the arguments are instances of some subclass of :class:`str`, + the return type is still plain :class:`str`. - A generic version of :class:`collections.abc.KeysView`. + At runtime, ``isinstance(x, T)`` will raise :exc:`TypeError`. In general, + :func:`isinstance` and :func:`issubclass` should not be used with types. -.. class:: ItemsView(MappingView, Generic[KT_co, VT_co]) + Type variables may be marked covariant or contravariant by passing + ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more + details. By default type variables are invariant. Alternatively, + a type variable may specify an upper bound using ``bound=``. + This means that an actual type substituted (explicitly or implicitly) + for the type variable must be a subclass of the boundary type, + see :pep:`484`. - A generic version of :class:`collections.abc.ItemsView`. +.. data:: AnyStr -.. class:: ValuesView(MappingView[VT_co]) + ``AnyStr`` is a type variable defined as + ``AnyStr = TypeVar('AnyStr', str, bytes)``. - A generic version of :class:`collections.abc.ValuesView`. + It is meant to be used for functions that may accept any kind of string + without allowing different kinds of strings to mix. For example:: -.. class:: Awaitable(Generic[T_co]) + def concat(a: AnyStr, b: AnyStr) -> AnyStr: + return a + b - A generic version of :class:`collections.abc.Awaitable`. + concat(u"foo", u"bar") # Ok, output has type 'unicode' + concat(b"foo", b"bar") # Ok, output has type 'bytes' + concat(u"foo", b"bar") # Error, cannot mix unicode and bytes - .. versionadded:: 3.5.2 +.. class:: Protocol(Generic) -.. class:: Coroutine(Awaitable[V_co], Generic[T_co T_contra, V_co]) + Base class for protocol classes. Protocol classes are defined like this:: - A generic version of :class:`collections.abc.Coroutine`. - The variance and order of type variables - correspond to those of :class:`Generator`, for example:: + class Proto(Protocol): + def meth(self) -> int: + ... - from typing import List, Coroutine - c = None # type: Coroutine[List[str], str, int] - ... - x = c.send('hi') # type: List[str] - async def bar() -> None: - x = await c # type: int + Such classes are primarily used with static type checkers that recognize + structural subtyping (static duck-typing), for example:: - .. versionadded:: 3.5.3 + class C: + def meth(self) -> int: + return 0 -.. class:: AsyncIterable(Generic[T_co]) + def func(x: Proto) -> int: + return x.meth() - A generic version of :class:`collections.abc.AsyncIterable`. + func(C()) # Passes static type check - .. versionadded:: 3.5.2 + See :pep:`544` for details. Protocol classes decorated with + :func:`runtime_checkable` (described later) act as simple-minded runtime + protocols that check only the presence of given attributes, ignoring their + type signatures. -.. class:: AsyncIterator(AsyncIterable[T_co]) + Protocol classes can be generic, for example:: - A generic version of :class:`collections.abc.AsyncIterator`. + class GenProto(Protocol[T]): + def meth(self) -> T: + ... - .. versionadded:: 3.5.2 + .. versionadded:: 3.8 -.. class:: ContextManager(Generic[T_co]) +.. decorator:: runtime_checkable - A generic version of :class:`contextlib.AbstractContextManager`. + Mark a protocol class as a runtime protocol. - .. versionadded:: 3.5.4 - .. versionadded:: 3.6.0 + Such a protocol can be used with :func:`isinstance` and :func:`issubclass`. + This raises :exc:`TypeError` when applied to a non-protocol class. This + allows a simple-minded structural check, very similar to "one trick ponies" + in :mod:`collections.abc` such as :class:`Iterable`. For example:: -.. class:: AsyncContextManager(Generic[T_co]) + @runtime_checkable + class Closable(Protocol): + def close(self): ... - A generic version of :class:`contextlib.AbstractAsyncContextManager`. + assert isinstance(open('/some/file'), Closable) - .. versionadded:: 3.5.4 - .. versionadded:: 3.6.2 + .. note:: -.. class:: Dict(dict, MutableMapping[KT, VT]) + :func:`runtime_checkable` will check only the presence of the required methods, + not their type signatures! For example, :class:`builtins.complex ` + implements :func:`__float__`, therefore it passes an :func:`issubclass` check + against :class:`SupportsFloat`. However, the ``complex.__float__`` method + exists only to raise a :class:`TypeError` with a more informative message. - A generic version of :class:`dict`. - Useful for annotating return types. To annotate arguments it is preferred - to use an abstract collection type such as :class:`Mapping`. + .. versionadded:: 3.8 - This type can be used as follows:: +Other special directives +"""""""""""""""""""""""" - def count_words(text: str) -> Dict[str, int]: - ... +These are not used in annotations. They are building blocks for declaring types. -.. class:: DefaultDict(collections.defaultdict, MutableMapping[KT, VT]) +.. class:: NamedTuple - A generic version of :class:`collections.defaultdict`. + Typed version of :func:`collections.namedtuple`. - .. versionadded:: 3.5.2 + Usage:: -.. class:: OrderedDict(collections.OrderedDict, MutableMapping[KT, VT]) + class Employee(NamedTuple): + name: str + id: int - A generic version of :class:`collections.OrderedDict`. + This is equivalent to:: - .. versionadded:: 3.7.2 + Employee = collections.namedtuple('Employee', ['name', 'id']) -.. class:: Counter(collections.Counter, Dict[T, int]) + To give a field a default value, you can assign to it in the class body:: - A generic version of :class:`collections.Counter`. + class Employee(NamedTuple): + name: str + id: int = 3 - .. versionadded:: 3.5.4 - .. versionadded:: 3.6.1 + employee = Employee('Guido') + assert employee.id == 3 -.. class:: ChainMap(collections.ChainMap, MutableMapping[KT, VT]) + Fields with a default value must come after any fields without a default. - A generic version of :class:`collections.ChainMap`. + The resulting class has an extra attribute ``__annotations__`` giving a + dict that maps the field names to the field types. (The field names are in + the ``_fields`` attribute and the default values are in the + ``_field_defaults`` attribute both of which are part of the namedtuple + API.) - .. versionadded:: 3.5.4 - .. versionadded:: 3.6.1 + ``NamedTuple`` subclasses can also have docstrings and methods:: -.. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) - - A generator can be annotated by the generic type - ``Generator[YieldType, SendType, ReturnType]``. For example:: - - def echo_round() -> Generator[int, float, str]: - sent = yield 0 - while sent >= 0: - sent = yield round(sent) - return 'Done' - - Note that unlike many other generics in the typing module, the ``SendType`` - of :class:`Generator` behaves contravariantly, not covariantly or - invariantly. - - If your generator will only yield values, set the ``SendType`` and - ``ReturnType`` to ``None``:: - - def infinite_stream(start: int) -> Generator[int, None, None]: - while True: - yield start - start += 1 - - Alternatively, annotate your generator as having a return type of - either ``Iterable[YieldType]`` or ``Iterator[YieldType]``:: - - def infinite_stream(start: int) -> Iterator[int]: - while True: - yield start - start += 1 - -.. class:: AsyncGenerator(AsyncIterator[T_co], Generic[T_co, T_contra]) - - An async generator can be annotated by the generic type - ``AsyncGenerator[YieldType, SendType]``. For example:: - - async def echo_round() -> AsyncGenerator[int, float]: - sent = yield 0 - while sent >= 0.0: - rounded = await round(sent) - sent = yield rounded - - Unlike normal generators, async generators cannot return a value, so there - is no ``ReturnType`` type parameter. As with :class:`Generator`, the - ``SendType`` behaves contravariantly. - - If your generator will only yield values, set the ``SendType`` to - ``None``:: - - async def infinite_stream(start: int) -> AsyncGenerator[int, None]: - while True: - yield start - start = await increment(start) - - Alternatively, annotate your generator as having a return type of - either ``AsyncIterable[YieldType]`` or ``AsyncIterator[YieldType]``:: - - async def infinite_stream(start: int) -> AsyncIterator[int]: - while True: - yield start - start = await increment(start) - - .. versionadded:: 3.6.1 - -.. class:: Text - - ``Text`` is an alias for ``str``. It is provided to supply a forward - compatible path for Python 2 code: in Python 2, ``Text`` is an alias for - ``unicode``. - - Use ``Text`` to indicate that a value must contain a unicode string in - a manner that is compatible with both Python 2 and Python 3:: - - def add_unicode_checkmark(text: Text) -> Text: - return text + u' \u2713' - - .. versionadded:: 3.5.2 - -.. class:: IO - TextIO - BinaryIO - - Generic type ``IO[AnyStr]`` and its subclasses ``TextIO(IO[str])`` - and ``BinaryIO(IO[bytes])`` - represent the types of I/O streams such as returned by - :func:`open`. - -.. class:: Pattern - Match - - These type aliases - correspond to the return types from :func:`re.compile` and - :func:`re.match`. These types (and the corresponding functions) - are generic in ``AnyStr`` and can be made specific by writing - ``Pattern[str]``, ``Pattern[bytes]``, ``Match[str]``, or - ``Match[bytes]``. - -.. class:: NamedTuple - - Typed version of :func:`collections.namedtuple`. - - Usage:: - - class Employee(NamedTuple): - name: str - id: int - - This is equivalent to:: - - Employee = collections.namedtuple('Employee', ['name', 'id']) - - To give a field a default value, you can assign to it in the class body:: - - class Employee(NamedTuple): - name: str - id: int = 3 - - employee = Employee('Guido') - assert employee.id == 3 - - Fields with a default value must come after any fields without a default. - - The resulting class has an extra attribute ``__annotations__`` giving a - dict that maps the field names to the field types. (The field names are in - the ``_fields`` attribute and the default values are in the - ``_field_defaults`` attribute both of which are part of the namedtuple - API.) - - ``NamedTuple`` subclasses can also have docstrings and methods:: - - class Employee(NamedTuple): - """Represents an employee.""" - name: str - id: int = 3 + class Employee(NamedTuple): + """Represents an employee.""" + name: str + id: int = 3 def __repr__(self) -> str: return f'' @@ -967,13 +982,23 @@ The module defines the following classes, functions and decorators: Removed the ``_field_types`` attribute in favor of the more standard ``__annotations__`` attribute which has the same information. +.. function:: NewType(name, tp) + + A helper function to indicate a distinct type to a typechecker, + see :ref:`distinct`. At runtime it returns a function that returns + its argument. Usage:: + + UserId = NewType('UserId', int) + first_user = UserId(1) + + .. versionadded:: 3.5.2 .. class:: TypedDict(dict) - A simple typed namespace. At runtime it is equivalent to - a plain :class:`dict`. + Special construct to add type hints to a dictionary. + At runtime it is a plain :class:`dict`. - ``TypedDict`` creates a dictionary type that expects all of its + ``TypedDict`` declares a dictionary type that expects all of its instances to have a certain set of keys, where each key is associated with a value of a consistent type. This expectation is not checked at runtime but is only enforced by type checkers. @@ -1014,474 +1039,658 @@ The module defines the following classes, functions and decorators: .. versionadded:: 3.8 -.. class:: ForwardRef +Generic concrete collections +---------------------------- - A class used for internal typing representation of string forward references. - For example, ``List["SomeClass"]`` is implicitly transformed into - ``List[ForwardRef("SomeClass")]``. This class should not be instantiated by - a user, but may be used by introspection tools. +Corresponding to built-in types +""""""""""""""""""""""""""""""" -.. function:: NewType(name, tp) +.. class:: Dict(dict, MutableMapping[KT, VT]) - A helper function to indicate a distinct type to a typechecker, - see :ref:`distinct`. At runtime it returns a function that returns - its argument. Usage:: + A generic version of :class:`dict`. + Useful for annotating return types. To annotate arguments it is preferred + to use an abstract collection type such as :class:`Mapping`. - UserId = NewType('UserId', int) - first_user = UserId(1) + This type can be used as follows:: - .. versionadded:: 3.5.2 + def count_words(text: str) -> Dict[str, int]: + ... -.. function:: cast(typ, val) + .. deprecated:: 3.9 + :class:`builtins.dict ` now supports ``[]``. See :pep:`585`. - Cast a value to a type. +.. class:: List(list, MutableSequence[T]) - This returns the value unchanged. To the type checker this - signals that the return value has the designated type, but at - runtime we intentionally don't check anything (we want this - to be as fast as possible). + Generic version of :class:`list`. + Useful for annotating return types. To annotate arguments it is preferred + to use an abstract collection type such as :class:`Sequence` or + :class:`Iterable`. -.. function:: get_type_hints(obj, globalns=None, localns=None, include_extras=False) + This type may be used as follows:: - Return a dictionary containing type hints for a function, method, module - or class object. + T = TypeVar('T', int, float) - This is often the same as ``obj.__annotations__``. In addition, - forward references encoded as string literals are handled by evaluating - them in ``globals`` and ``locals`` namespaces. If necessary, - ``Optional[t]`` is added for function and method annotations if a default - value equal to ``None`` is set. For a class ``C``, return - a dictionary constructed by merging all the ``__annotations__`` along - ``C.__mro__`` in reverse order. + def vec2(x: T, y: T) -> List[T]: + return [x, y] - The function recursively replaces all ``Annotated[T, ...]`` with ``T``, - unless ``include_extras`` is set to ``True`` (see :class:`Annotated` for - more information). For example:: + def keep_positives(vector: Sequence[T]) -> List[T]: + return [item for item in vector if item > 0] - class Student(NamedTuple): - name: Annotated[str, 'some marker'] + .. deprecated:: 3.9 + :class:`builtins.list ` now supports ``[]``. See :pep:`585`. - get_type_hints(Student) == {'name': str} - get_type_hints(Student, include_extras=False) == {'name': str} - get_type_hints(Student, include_extras=True) == { - 'name': Annotated[str, 'some marker'] - } +.. class:: Set(set, MutableSet[T]) - .. versionchanged:: 3.9 - Added ``include_extras`` parameter as part of :pep:`593`. + A generic version of :class:`builtins.set `. + Useful for annotating return types. To annotate arguments it is preferred + to use an abstract collection type such as :class:`AbstractSet`. -.. function:: get_origin(tp) -.. function:: get_args(tp) + .. deprecated:: 3.9 + :class:`builtins.set ` now supports ``[]``. See :pep:`585`. - Provide basic introspection for generic types and special typing forms. +.. class:: FrozenSet(frozenset, AbstractSet[T_co]) - For a typing object of the form ``X[Y, Z, ...]`` these functions return - ``X`` and ``(Y, Z, ...)``. If ``X`` is a generic alias for a builtin or - :mod:`collections` class, it gets normalized to the original class. - For unsupported objects return ``None`` and ``()`` correspondingly. - Examples:: + A generic version of :class:`builtins.frozenset `. - assert get_origin(Dict[str, int]) is dict - assert get_args(Dict[int, str]) == (int, str) + .. deprecated:: 3.9 + :class:`builtins.frozenset ` now supports ``[]``. See :pep:`585`. - assert get_origin(Union[int, str]) is Union - assert get_args(Union[int, str]) == (int, str) +.. note:: :data:`Tuple` is a special form. - .. versionadded:: 3.8 +Corresponding to types in :mod:`collections` +"""""""""""""""""""""""""""""""""""""""""""" -.. decorator:: overload +.. class:: DefaultDict(collections.defaultdict, MutableMapping[KT, VT]) - The ``@overload`` decorator allows describing functions and methods - that support multiple different combinations of argument types. A series - of ``@overload``-decorated definitions must be followed by exactly one - non-``@overload``-decorated definition (for the same function/method). - The ``@overload``-decorated definitions are for the benefit of the - type checker only, since they will be overwritten by the - non-``@overload``-decorated definition, while the latter is used at - runtime but should be ignored by a type checker. At runtime, calling - a ``@overload``-decorated function directly will raise - :exc:`NotImplementedError`. An example of overload that gives a more - precise type than can be expressed using a union or a type variable:: + A generic version of :class:`collections.defaultdict`. - @overload - def process(response: None) -> None: - ... - @overload - def process(response: int) -> Tuple[int, str]: - ... - @overload - def process(response: bytes) -> str: - ... - def process(response): - + .. versionadded:: 3.5.2 - See :pep:`484` for details and comparison with other typing semantics. + .. deprecated:: 3.9 + :class:`collections.defaultdict` now supports ``[]``. See :pep:`585`. -.. decorator:: final +.. class:: OrderedDict(collections.OrderedDict, MutableMapping[KT, VT]) - A decorator to indicate to type checkers that the decorated method - cannot be overridden, and the decorated class cannot be subclassed. - For example:: + A generic version of :class:`collections.OrderedDict`. - class Base: - @final - def done(self) -> None: - ... - class Sub(Base): - def done(self) -> None: # Error reported by type checker - ... + .. versionadded:: 3.7.2 - @final - class Leaf: - ... - class Other(Leaf): # Error reported by type checker - ... + .. deprecated:: 3.9 + :class:`collections.OrderedDict` now supports ``[]``. See :pep:`585`. - There is no runtime checking of these properties. See :pep:`591` for - more details. +.. class:: ChainMap(collections.ChainMap, MutableMapping[KT, VT]) - .. versionadded:: 3.8 + A generic version of :class:`collections.ChainMap`. -.. decorator:: no_type_check + .. versionadded:: 3.5.4 + .. versionadded:: 3.6.1 - Decorator to indicate that annotations are not type hints. + .. deprecated:: 3.9 + :class:`collections.ChainMap` now supports ``[]``. See :pep:`585`. - This works as class or function :term:`decorator`. With a class, it - applies recursively to all methods defined in that class (but not - to methods defined in its superclasses or subclasses). +.. class:: Counter(collections.Counter, Dict[T, int]) - This mutates the function(s) in place. + A generic version of :class:`collections.Counter`. -.. decorator:: no_type_check_decorator + .. versionadded:: 3.5.4 + .. versionadded:: 3.6.1 - Decorator to give another decorator the :func:`no_type_check` effect. + .. deprecated:: 3.9 + :class:`collections.Counter` now supports ``[]``. See :pep:`585`. - This wraps the decorator with something that wraps the decorated - function in :func:`no_type_check`. +.. class:: Deque(deque, MutableSequence[T]) -.. decorator:: type_check_only + A generic version of :class:`collections.deque`. - Decorator to mark a class or function to be unavailable at runtime. + .. versionadded:: 3.5.4 + .. versionadded:: 3.6.1 - This decorator is itself not available at runtime. It is mainly - intended to mark classes that are defined in type stub files if - an implementation returns an instance of a private class:: + .. deprecated:: 3.9 + :class:`collections.deque` now supports ``[]``. See :pep:`585`. - @type_check_only - class Response: # private or not available at runtime - code: int - def get_header(self, name: str) -> str: ... +Other concrete types +"""""""""""""""""""" - def fetch_response() -> Response: ... +.. class:: IO + TextIO + BinaryIO - Note that returning instances of private classes is not recommended. - It is usually preferable to make such classes public. + Generic type ``IO[AnyStr]`` and its subclasses ``TextIO(IO[str])`` + and ``BinaryIO(IO[bytes])`` + represent the types of I/O streams such as returned by + :func:`open`. These types are also in the ``typing.io`` namespace. -.. decorator:: runtime_checkable +.. class:: Pattern + Match - Mark a protocol class as a runtime protocol. + These type aliases + correspond to the return types from :func:`re.compile` and + :func:`re.match`. These types (and the corresponding functions) + are generic in ``AnyStr`` and can be made specific by writing + ``Pattern[str]``, ``Pattern[bytes]``, ``Match[str]``, or + ``Match[bytes]``. These types are also in the ``typing.re`` namespace. - Such a protocol can be used with :func:`isinstance` and :func:`issubclass`. - This raises :exc:`TypeError` when applied to a non-protocol class. This - allows a simple-minded structural check, very similar to "one trick ponies" - in :mod:`collections.abc` such as :class:`Iterable`. For example:: + .. deprecated:: 3.9 + Classes ``Pattern`` and ``Match`` from :mod:`re` now support ``[]``. See :pep:`585`. - @runtime_checkable - class Closable(Protocol): - def close(self): ... +.. class:: Text - assert isinstance(open('/some/file'), Closable) + ``Text`` is an alias for ``str``. It is provided to supply a forward + compatible path for Python 2 code: in Python 2, ``Text`` is an alias for + ``unicode``. - **Warning:** this will check only the presence of the required methods, - not their type signatures! + Use ``Text`` to indicate that a value must contain a unicode string in + a manner that is compatible with both Python 2 and Python 3:: - .. versionadded:: 3.8 + def add_unicode_checkmark(text: Text) -> Text: + return text + u' \u2713' -.. data:: Any + .. versionadded:: 3.5.2 - Special type indicating an unconstrained type. +Abstract Base Classes +--------------------- - * Every type is compatible with :data:`Any`. - * :data:`Any` is compatible with every type. +Corresponding to collections in :mod:`collections.abc` +"""""""""""""""""""""""""""""""""""""""""""""""""""""" -.. data:: NoReturn +.. class:: AbstractSet(Sized, Collection[T_co]) - Special type indicating that a function never returns. - For example:: + A generic version of :class:`collections.abc.Set`. - from typing import NoReturn + .. deprecated:: 3.9 + :class:`collections.abc.Set` now supports ``[]``. See :pep:`585`. - def stop() -> NoReturn: - raise RuntimeError('no way') +.. class:: ByteString(Sequence[int]) - .. versionadded:: 3.5.4 - .. versionadded:: 3.6.2 + A generic version of :class:`collections.abc.ByteString`. -.. data:: Union + This type represents the types :class:`bytes`, :class:`bytearray`, + and :class:`memoryview` of byte sequences. - Union type; ``Union[X, Y]`` means either X or Y. + As a shorthand for this type, :class:`bytes` can be used to + annotate arguments of any of the types mentioned above. - To define a union, use e.g. ``Union[int, str]``. Details: + .. deprecated:: 3.9 + :class:`collections.abc.ByteString` now supports ``[]``. See :pep:`585`. - * The arguments must be types and there must be at least one. +.. class:: Collection(Sized, Iterable[T_co], Container[T_co]) - * Unions of unions are flattened, e.g.:: + A generic version of :class:`collections.abc.Collection` - Union[Union[int, str], float] == Union[int, str, float] + .. versionadded:: 3.6.0 - * Unions of a single argument vanish, e.g.:: + .. deprecated:: 3.9 + :class:`collections.abc.Collection` now supports ``[]``. See :pep:`585`. - Union[int] == int # The constructor actually returns int +.. class:: Container(Generic[T_co]) - * Redundant arguments are skipped, e.g.:: + A generic version of :class:`collections.abc.Container`. - Union[int, str, int] == Union[int, str] + .. deprecated:: 3.9 + :class:`collections.abc.Container` now supports ``[]``. See :pep:`585`. - * When comparing unions, the argument order is ignored, e.g.:: +.. class:: ItemsView(MappingView, Generic[KT_co, VT_co]) - Union[int, str] == Union[str, int] + A generic version of :class:`collections.abc.ItemsView`. - * You cannot subclass or instantiate a union. + .. deprecated:: 3.9 + :class:`collections.abc.ItemsView` now supports ``[]``. See :pep:`585`. - * You cannot write ``Union[X][Y]``. +.. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co]) - * You can use ``Optional[X]`` as a shorthand for ``Union[X, None]``. + A generic version of :class:`collections.abc.KeysView`. - .. versionchanged:: 3.7 - Don't remove explicit subclasses from unions at runtime. + .. deprecated:: 3.9 + :class:`collections.abc.KeysView` now supports ``[]``. See :pep:`585`. -.. data:: Optional +.. class:: Mapping(Sized, Collection[KT], Generic[VT_co]) - Optional type. + A generic version of :class:`collections.abc.Mapping`. + This type can be used as follows:: - ``Optional[X]`` is equivalent to ``Union[X, None]``. + def get_position_in_index(word_list: Mapping[str, int], word: str) -> int: + return word_list[word] - Note that this is not the same concept as an optional argument, - which is one that has a default. An optional argument with a - default does not require the ``Optional`` qualifier on its type - annotation just because it is optional. For example:: + .. deprecated:: 3.9 + :class:`collections.abc.Mapping` now supports ``[]``. See :pep:`585`. - def foo(arg: int = 0) -> None: - ... +.. class:: MappingView(Sized, Iterable[T_co]) - On the other hand, if an explicit value of ``None`` is allowed, the - use of ``Optional`` is appropriate, whether the argument is optional - or not. For example:: + A generic version of :class:`collections.abc.MappingView`. - def foo(arg: Optional[int] = None) -> None: - ... + .. deprecated:: 3.9 + :class:`collections.abc.MappingView` now supports ``[]``. See :pep:`585`. -.. data:: Tuple +.. class:: MutableMapping(Mapping[KT, VT]) - Tuple type; ``Tuple[X, Y]`` is the type of a tuple of two items - with the first item of type X and the second of type Y. The type of - the empty tuple can be written as ``Tuple[()]``. + A generic version of :class:`collections.abc.MutableMapping`. - Example: ``Tuple[T1, T2]`` is a tuple of two elements corresponding - to type variables T1 and T2. ``Tuple[int, float, str]`` is a tuple - of an int, a float and a string. + .. deprecated:: 3.9 + :class:`collections.abc.MutableMapping` now supports ``[]``. See :pep:`585`. - To specify a variable-length tuple of homogeneous type, - use literal ellipsis, e.g. ``Tuple[int, ...]``. A plain :data:`Tuple` - is equivalent to ``Tuple[Any, ...]``, and in turn to :class:`tuple`. +.. class:: MutableSequence(Sequence[T]) -.. data:: Callable + A generic version of :class:`collections.abc.MutableSequence`. - Callable type; ``Callable[[int], str]`` is a function of (int) -> str. + .. deprecated:: 3.9 + :class:`collections.abc.MutableSequence` now supports ``[]``. See :pep:`585`. - The subscription syntax must always be used with exactly two - values: the argument list and the return type. The argument list - must be a list of types or an ellipsis; the return type must be - a single type. +.. class:: MutableSet(AbstractSet[T]) - There is no syntax to indicate optional or keyword arguments; - such function types are rarely used as callback types. - ``Callable[..., ReturnType]`` (literal ellipsis) can be used to - type hint a callable taking any number of arguments and returning - ``ReturnType``. A plain :data:`Callable` is equivalent to - ``Callable[..., Any]``, and in turn to - :class:`collections.abc.Callable`. + A generic version of :class:`collections.abc.MutableSet`. -.. data:: Literal + .. deprecated:: 3.9 + :class:`collections.abc.MutableSet` now supports ``[]``. See :pep:`585`. - A type that can be used to indicate to type checkers that the - corresponding variable or function parameter has a value equivalent to - the provided literal (or one of several literals). For example:: +.. class:: Sequence(Reversible[T_co], Collection[T_co]) - def validate_simple(data: Any) -> Literal[True]: # always returns True - ... + A generic version of :class:`collections.abc.Sequence`. - MODE = Literal['r', 'rb', 'w', 'wb'] - def open_helper(file: str, mode: MODE) -> str: - ... + .. deprecated:: 3.9 + :class:`collections.abc.Sequence` now supports ``[]``. See :pep:`585`. - open_helper('/some/path', 'r') # Passes type check - open_helper('/other/path', 'typo') # Error in type checker +.. class:: ValuesView(MappingView[VT_co]) - ``Literal[...]`` cannot be subclassed. At runtime, an arbitrary value - is allowed as type argument to ``Literal[...]``, but type checkers may - impose restrictions. See :pep:`586` for more details about literal types. + A generic version of :class:`collections.abc.ValuesView`. - .. versionadded:: 3.8 + .. deprecated:: 3.9 + :class:`collections.abc.ValuesView` now supports ``[]``. See :pep:`585`. -.. data:: ClassVar +Corresponding to other types in :mod:`collections.abc` +"""""""""""""""""""""""""""""""""""""""""""""""""""""" - Special type construct to mark class variables. +.. class:: Iterable(Generic[T_co]) - As introduced in :pep:`526`, a variable annotation wrapped in ClassVar - indicates that a given attribute is intended to be used as a class variable - and should not be set on instances of that class. Usage:: + A generic version of :class:`collections.abc.Iterable`. - class Starship: - stats: ClassVar[Dict[str, int]] = {} # class variable - damage: int = 10 # instance variable + .. deprecated:: 3.9 + :class:`collections.abc.Iterable` now supports ``[]``. See :pep:`585`. - :data:`ClassVar` accepts only types and cannot be further subscribed. +.. class:: Iterator(Iterable[T_co]) - :data:`ClassVar` is not a class itself, and should not - be used with :func:`isinstance` or :func:`issubclass`. - :data:`ClassVar` does not change Python runtime behavior, but - it can be used by third-party type checkers. For example, a type checker - might flag the following code as an error:: + A generic version of :class:`collections.abc.Iterator`. - enterprise_d = Starship(3000) - enterprise_d.stats = {} # Error, setting class variable on instance - Starship.stats = {} # This is OK + .. deprecated:: 3.9 + :class:`collections.abc.Iterator` now supports ``[]``. See :pep:`585`. - .. versionadded:: 3.5.3 +.. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) -.. data:: Final + A generator can be annotated by the generic type + ``Generator[YieldType, SendType, ReturnType]``. For example:: - A special typing construct to indicate to type checkers that a name - cannot be re-assigned or overridden in a subclass. For example:: + def echo_round() -> Generator[int, float, str]: + sent = yield 0 + while sent >= 0: + sent = yield round(sent) + return 'Done' - MAX_SIZE: Final = 9000 - MAX_SIZE += 1 # Error reported by type checker + Note that unlike many other generics in the typing module, the ``SendType`` + of :class:`Generator` behaves contravariantly, not covariantly or + invariantly. - class Connection: - TIMEOUT: Final[int] = 10 + If your generator will only yield values, set the ``SendType`` and + ``ReturnType`` to ``None``:: - class FastConnector(Connection): - TIMEOUT = 1 # Error reported by type checker + def infinite_stream(start: int) -> Generator[int, None, None]: + while True: + yield start + start += 1 - There is no runtime checking of these properties. See :pep:`591` for - more details. + Alternatively, annotate your generator as having a return type of + either ``Iterable[YieldType]`` or ``Iterator[YieldType]``:: - .. versionadded:: 3.8 + def infinite_stream(start: int) -> Iterator[int]: + while True: + yield start + start += 1 -.. data:: AnyStr + .. deprecated:: 3.9 + :class:`collections.abc.Generator` now supports ``[]``. See :pep:`585`. - ``AnyStr`` is a type variable defined as - ``AnyStr = TypeVar('AnyStr', str, bytes)``. +.. class:: Hashable - It is meant to be used for functions that may accept any kind of string - without allowing different kinds of strings to mix. For example:: + An alias to :class:`collections.abc.Hashable` - def concat(a: AnyStr, b: AnyStr) -> AnyStr: - return a + b +.. class:: Reversible(Iterable[T_co]) - concat(u"foo", u"bar") # Ok, output has type 'unicode' - concat(b"foo", b"bar") # Ok, output has type 'bytes' - concat(u"foo", b"bar") # Error, cannot mix unicode and bytes + A generic version of :class:`collections.abc.Reversible`. -.. data:: TYPE_CHECKING + .. deprecated:: 3.9 + :class:`collections.abc.Reversible` now supports ``[]``. See :pep:`585`. - A special constant that is assumed to be ``True`` by 3rd party static - type checkers. It is ``False`` at runtime. Usage:: +.. class:: Sized - if TYPE_CHECKING: - import expensive_mod + An alias to :class:`collections.abc.Sized` - def fun(arg: 'expensive_mod.SomeType') -> None: - local_var: expensive_mod.AnotherType = other_fun() +Asynchronous programming +"""""""""""""""""""""""" - Note that the first type annotation must be enclosed in quotes, making it a - "forward reference", to hide the ``expensive_mod`` reference from the - interpreter runtime. Type annotations for local variables are not - evaluated, so the second annotation does not need to be enclosed in quotes. +.. class:: Coroutine(Awaitable[V_co], Generic[T_co T_contra, V_co]) + + A generic version of :class:`collections.abc.Coroutine`. + The variance and order of type variables + correspond to those of :class:`Generator`, for example:: + + from typing import List, Coroutine + c = None # type: Coroutine[List[str], str, int] + ... + x = c.send('hi') # type: List[str] + async def bar() -> None: + x = await c # type: int + + .. versionadded:: 3.5.3 + + .. deprecated:: 3.9 + :class:`collections.abc.Coroutine` now supports ``[]``. See :pep:`585`. + +.. class:: AsyncGenerator(AsyncIterator[T_co], Generic[T_co, T_contra]) + + An async generator can be annotated by the generic type + ``AsyncGenerator[YieldType, SendType]``. For example:: + + async def echo_round() -> AsyncGenerator[int, float]: + sent = yield 0 + while sent >= 0.0: + rounded = await round(sent) + sent = yield rounded + + Unlike normal generators, async generators cannot return a value, so there + is no ``ReturnType`` type parameter. As with :class:`Generator`, the + ``SendType`` behaves contravariantly. + + If your generator will only yield values, set the ``SendType`` to + ``None``:: + + async def infinite_stream(start: int) -> AsyncGenerator[int, None]: + while True: + yield start + start = await increment(start) + + Alternatively, annotate your generator as having a return type of + either ``AsyncIterable[YieldType]`` or ``AsyncIterator[YieldType]``:: + + async def infinite_stream(start: int) -> AsyncIterator[int]: + while True: + yield start + start = await increment(start) + + .. versionadded:: 3.6.1 + + .. deprecated:: 3.9 + :class:`collections.abc.AsyncGenerator` now supports ``[]``. See :pep:`585`. + +.. class:: AsyncIterable(Generic[T_co]) + + A generic version of :class:`collections.abc.AsyncIterable`. .. versionadded:: 3.5.2 -.. data:: Annotated + .. deprecated:: 3.9 + :class:`collections.abc.AsyncIterable` now supports ``[]``. See :pep:`585`. - A type, introduced in :pep:`593` (``Flexible function and variable - annotations``), to decorate existing types with context-specific metadata - (possibly multiple pieces of it, as ``Annotated`` is variadic). - Specifically, a type ``T`` can be annotated with metadata ``x`` via the - typehint ``Annotated[T, x]``. This metadata can be used for either static - analysis or at runtime. If a library (or tool) encounters a typehint - ``Annotated[T, x]`` and has no special logic for metadata ``x``, it - should ignore it and simply treat the type as ``T``. Unlike the - ``no_type_check`` functionality that currently exists in the ``typing`` - module which completely disables typechecking annotations on a function - or a class, the ``Annotated`` type allows for both static typechecking - of ``T`` (e.g., via mypy or Pyre, which can safely ignore ``x``) - together with runtime access to ``x`` within a specific application. +.. class:: AsyncIterator(AsyncIterable[T_co]) - Ultimately, the responsibility of how to interpret the annotations (if - at all) is the responsibility of the tool or library encountering the - ``Annotated`` type. A tool or library encountering an ``Annotated`` type - can scan through the annotations to determine if they are of interest - (e.g., using ``isinstance()``). + A generic version of :class:`collections.abc.AsyncIterator`. - When a tool or a library does not support annotations or encounters an - unknown annotation it should just ignore it and treat annotated type as - the underlying type. + .. versionadded:: 3.5.2 - It's up to the tool consuming the annotations to decide whether the - client is allowed to have several annotations on one type and how to - merge those annotations. + .. deprecated:: 3.9 + :class:`collections.abc.AsyncIterator` now supports ``[]``. See :pep:`585`. - Since the ``Annotated`` type allows you to put several annotations of - the same (or different) type(s) on any node, the tools or libraries - consuming those annotations are in charge of dealing with potential - duplicates. For example, if you are doing value range analysis you might - allow this:: +.. class:: Awaitable(Generic[T_co]) - T1 = Annotated[int, ValueRange(-10, 5)] - T2 = Annotated[T1, ValueRange(-20, 3)] + A generic version of :class:`collections.abc.Awaitable`. - Passing ``include_extras=True`` to :func:`get_type_hints` lets one - access the extra annotations at runtime. + .. versionadded:: 3.5.2 - The details of the syntax: + .. deprecated:: 3.9 + :class:`collections.abc.Awaitable` now supports ``[]``. See :pep:`585`. - * The first argument to ``Annotated`` must be a valid type - * Multiple type annotations are supported (``Annotated`` supports variadic - arguments):: +Context manager types +""""""""""""""""""""" - Annotated[int, ValueRange(3, 10), ctype("char")] +.. class:: ContextManager(Generic[T_co]) - * ``Annotated`` must be called with at least two arguments ( - ``Annotated[int]`` is not valid) + A generic version of :class:`contextlib.AbstractContextManager`. - * The order of the annotations is preserved and matters for equality - checks:: + .. versionadded:: 3.5.4 + .. versionadded:: 3.6.0 - Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[ - int, ctype("char"), ValueRange(3, 10) - ] + .. deprecated:: 3.9 + :class:`collections.contextlib.AbstractContextManager` now supports ``[]``. See :pep:`585`. - * Nested ``Annotated`` types are flattened, with metadata ordered - starting with the innermost annotation:: +.. class:: AsyncContextManager(Generic[T_co]) - Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[ - int, ValueRange(3, 10), ctype("char") - ] + A generic version of :class:`contextlib.AbstractAsyncContextManager`. - * Duplicated annotations are not removed:: + .. versionadded:: 3.5.4 + .. versionadded:: 3.6.2 - Annotated[int, ValueRange(3, 10)] != Annotated[ - int, ValueRange(3, 10), ValueRange(3, 10) - ] + .. deprecated:: 3.9 + :class:`collections.contextlib.AbstractAsyncContextManager` now supports ``[]``. See :pep:`585`. - * ``Annotated`` can be used with nested and generic aliases:: +Protocols +--------- - T = TypeVar('T') - Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] - V = Vec[int] +These protocols are decorated with :func:`runtime_checkable`. - V == Annotated[List[Tuple[int, int]], MaxLen(10)] +.. class:: SupportsAbs + + An ABC with one abstract method ``__abs__`` that is covariant + in its return type. + +.. class:: SupportsBytes + + An ABC with one abstract method ``__bytes__``. + +.. class:: SupportsComplex + + An ABC with one abstract method ``__complex__``. + +.. class:: SupportsFloat + + An ABC with one abstract method ``__float__``. + +.. class:: SupportsIndex + + An ABC with one abstract method ``__index__``. + + .. versionadded:: 3.8 + +.. class:: SupportsInt + + An ABC with one abstract method ``__int__``. + +.. class:: SupportsRound + + An ABC with one abstract method ``__round__`` + that is covariant in its return type. + +Functions and decorators +------------------------ + +.. function:: cast(typ, val) + + Cast a value to a type. + + This returns the value unchanged. To the type checker this + signals that the return value has the designated type, but at + runtime we intentionally don't check anything (we want this + to be as fast as possible). + +.. decorator:: overload + + The ``@overload`` decorator allows describing functions and methods + that support multiple different combinations of argument types. A series + of ``@overload``-decorated definitions must be followed by exactly one + non-``@overload``-decorated definition (for the same function/method). + The ``@overload``-decorated definitions are for the benefit of the + type checker only, since they will be overwritten by the + non-``@overload``-decorated definition, while the latter is used at + runtime but should be ignored by a type checker. At runtime, calling + a ``@overload``-decorated function directly will raise + :exc:`NotImplementedError`. An example of overload that gives a more + precise type than can be expressed using a union or a type variable:: + + @overload + def process(response: None) -> None: + ... + @overload + def process(response: int) -> Tuple[int, str]: + ... + @overload + def process(response: bytes) -> str: + ... + def process(response): + + + See :pep:`484` for details and comparison with other typing semantics. + +.. decorator:: final + + A decorator to indicate to type checkers that the decorated method + cannot be overridden, and the decorated class cannot be subclassed. + For example:: + + class Base: + @final + def done(self) -> None: + ... + class Sub(Base): + def done(self) -> None: # Error reported by type checker + ... + + @final + class Leaf: + ... + class Other(Leaf): # Error reported by type checker + ... + + There is no runtime checking of these properties. See :pep:`591` for + more details. + + .. versionadded:: 3.8 + +.. decorator:: no_type_check + + Decorator to indicate that annotations are not type hints. + + This works as class or function :term:`decorator`. With a class, it + applies recursively to all methods defined in that class (but not + to methods defined in its superclasses or subclasses). + + This mutates the function(s) in place. + +.. decorator:: no_type_check_decorator + + Decorator to give another decorator the :func:`no_type_check` effect. + + This wraps the decorator with something that wraps the decorated + function in :func:`no_type_check`. + +.. decorator:: type_check_only + + Decorator to mark a class or function to be unavailable at runtime. + + This decorator is itself not available at runtime. It is mainly + intended to mark classes that are defined in type stub files if + an implementation returns an instance of a private class:: + + @type_check_only + class Response: # private or not available at runtime + code: int + def get_header(self, name: str) -> str: ... + + def fetch_response() -> Response: ... + + Note that returning instances of private classes is not recommended. + It is usually preferable to make such classes public. + +Introspection helpers +--------------------- + +.. function:: get_type_hints(obj, globalns=None, localns=None, include_extras=False) + + Return a dictionary containing type hints for a function, method, module + or class object. + + This is often the same as ``obj.__annotations__``. In addition, + forward references encoded as string literals are handled by evaluating + them in ``globals`` and ``locals`` namespaces. If necessary, + ``Optional[t]`` is added for function and method annotations if a default + value equal to ``None`` is set. For a class ``C``, return + a dictionary constructed by merging all the ``__annotations__`` along + ``C.__mro__`` in reverse order. + + The function recursively replaces all ``Annotated[T, ...]`` with ``T``, + unless ``include_extras`` is set to ``True`` (see :class:`Annotated` for + more information). For example:: + + class Student(NamedTuple): + name: Annotated[str, 'some marker'] + + get_type_hints(Student) == {'name': str} + get_type_hints(Student, include_extras=False) == {'name': str} + get_type_hints(Student, include_extras=True) == { + 'name': Annotated[str, 'some marker'] + } + + .. versionchanged:: 3.9 + Added ``include_extras`` parameter as part of :pep:`593`. + +.. function:: get_args(tp) +.. function:: get_origin(tp) + + Provide basic introspection for generic types and special typing forms. + + For a typing object of the form ``X[Y, Z, ...]`` these functions return + ``X`` and ``(Y, Z, ...)``. If ``X`` is a generic alias for a builtin or + :mod:`collections` class, it gets normalized to the original class. + For unsupported objects return ``None`` and ``()`` correspondingly. + Examples:: + + assert get_origin(Dict[str, int]) is dict + assert get_args(Dict[int, str]) == (int, str) + + assert get_origin(Union[int, str]) is Union + assert get_args(Union[int, str]) == (int, str) + + .. versionadded:: 3.8 + +.. class:: ForwardRef + + A class used for internal typing representation of string forward references. + For example, ``List["SomeClass"]`` is implicitly transformed into + ``List[ForwardRef("SomeClass")]``. This class should not be instantiated by + a user, but may be used by introspection tools. + +Constant +-------- + +.. data:: TYPE_CHECKING + + A special constant that is assumed to be ``True`` by 3rd party static + type checkers. It is ``False`` at runtime. Usage:: + + if TYPE_CHECKING: + import expensive_mod + + def fun(arg: 'expensive_mod.SomeType') -> None: + local_var: expensive_mod.AnotherType = other_fun() + + The first type annotation must be enclosed in quotes, making it a + "forward reference", to hide the ``expensive_mod`` reference from the + interpreter runtime. Type annotations for local variables are not + evaluated, so the second annotation does not need to be enclosed in quotes. + + .. note:: + + If ``from __future__ import annotations`` is used in Python 3.7 or later, + annotations are not evaluated at function definition time. + Instead, the are stored as strings in ``__annotations__``, + This makes it unnecessary to use quotes around the annotation. + (see :pep:`563`). + + .. versionadded:: 3.5.2 - .. versionadded:: 3.9 diff --git a/Misc/NEWS.d/next/Documentation/2020-07-21-15-23-30.bpo-40979.pLA8rO.rst b/Misc/NEWS.d/next/Documentation/2020-07-21-15-23-30.bpo-40979.pLA8rO.rst new file mode 100644 index 0000000000000..b0ca4327ad61a --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-07-21-15-23-30.bpo-40979.pLA8rO.rst @@ -0,0 +1 @@ +Refactored typing.rst, arranging more than 70 classes, functions, and decorators into new sub-sections. \ No newline at end of file From webhook-mailer at python.org Sun Aug 2 22:13:11 2020 From: webhook-mailer at python.org (Nathan M) Date: Mon, 03 Aug 2020 02:13:11 -0000 Subject: [Python-checkins] bpo-41424: Remove extra words in Tkinter-Packer documentation (GH-21707) Message-ID: https://github.com/python/cpython/commit/ecaf949cc487887883c14dff7a96e09ac9404994 commit: ecaf949cc487887883c14dff7a96e09ac9404994 branch: master author: Nathan M committer: GitHub date: 2020-08-02T22:13:03-04:00 summary: bpo-41424: Remove extra words in Tkinter-Packer documentation (GH-21707) files: M Doc/library/tkinter.rst diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index 2dc44ad36a7f7..3d90b4be17c97 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -464,12 +464,11 @@ The Packer .. index:: single: packing (widgets) The packer is one of Tk's geometry-management mechanisms. Geometry managers -are used to specify the relative positioning of the positioning of widgets -within their container - their mutual *master*. In contrast to the more -cumbersome *placer* (which is used less commonly, and we do not cover here), the -packer takes qualitative relationship specification - *above*, *to the left of*, -*filling*, etc - and works everything out to determine the exact placement -coordinates for you. +are used to specify the relative positioning of widgets within their container - +their mutual *master*. In contrast to the more cumbersome *placer* (which is +used less commonly, and we do not cover here), the packer takes qualitative +relationship specification - *above*, *to the left of*, *filling*, etc - and +works everything out to determine the exact placement coordinates for you. The size of any *master* widget is determined by the size of the "slave widgets" inside. The packer is used to control where slave widgets appear inside the From webhook-mailer at python.org Sun Aug 2 23:19:07 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 03 Aug 2020 03:19:07 -0000 Subject: [Python-checkins] bpo-41424: Remove extra words in Tkinter-Packer documentation (GH-21707) Message-ID: https://github.com/python/cpython/commit/45d37cbb04279a52c63dd180aacda7d35e02a0ef commit: 45d37cbb04279a52c63dd180aacda7d35e02a0ef branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-02T20:18:59-07:00 summary: bpo-41424: Remove extra words in Tkinter-Packer documentation (GH-21707) (cherry picked from commit ecaf949cc487887883c14dff7a96e09ac9404994) Co-authored-by: Nathan M files: M Doc/library/tkinter.rst diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index e1fc051d9597c..a04926b44612e 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -464,12 +464,11 @@ The Packer .. index:: single: packing (widgets) The packer is one of Tk's geometry-management mechanisms. Geometry managers -are used to specify the relative positioning of the positioning of widgets -within their container - their mutual *master*. In contrast to the more -cumbersome *placer* (which is used less commonly, and we do not cover here), the -packer takes qualitative relationship specification - *above*, *to the left of*, -*filling*, etc - and works everything out to determine the exact placement -coordinates for you. +are used to specify the relative positioning of widgets within their container - +their mutual *master*. In contrast to the more cumbersome *placer* (which is +used less commonly, and we do not cover here), the packer takes qualitative +relationship specification - *above*, *to the left of*, *filling*, etc - and +works everything out to determine the exact placement coordinates for you. The size of any *master* widget is determined by the size of the "slave widgets" inside. The packer is used to control where slave widgets appear inside the From webhook-mailer at python.org Mon Aug 3 00:03:57 2020 From: webhook-mailer at python.org (Ankit Chandawala) Date: Mon, 03 Aug 2020 04:03:57 -0000 Subject: [Python-checkins] bpo-41425: Make tkinter doc example runnable (GH-21706) Message-ID: https://github.com/python/cpython/commit/c36dbac588e1d99975f285a874bb20e9f5040af4 commit: c36dbac588e1d99975f285a874bb20e9f5040af4 branch: master author: Ankit Chandawala committer: GitHub date: 2020-08-03T00:03:48-04:00 summary: bpo-41425: Make tkinter doc example runnable (GH-21706) Co-authored-by: Ankit Chandawala Co-authored-by: Terry Jan Reedy files: A Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst M Doc/library/tkinter.rst diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index 3d90b4be17c97..9f954255c8b30 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -541,31 +541,35 @@ the variable, with no further intervention on your part. For example:: - class App(Frame): - def __init__(self, master=None): + import tkinter as tk + + class App(tk.Frame): + def __init__(self, master): super().__init__(master) self.pack() - self.entrythingy = Entry() + self.entrythingy = tk.Entry() self.entrythingy.pack() - # here is the application variable - self.contents = StringVar() - # set it to some value + # Create the application variable. + self.contents = tk.StringVar() + # Set it to some value. self.contents.set("this is a variable") - # tell the entry widget to watch this variable + # Tell the entry widget to watch this variable. self.entrythingy["textvariable"] = self.contents - # and here we get a callback when the user hits return. - # we will have the program print out the value of the - # application variable when the user hits return + # Define a callback for when the user hits return. + # It prints the current value of the variable. self.entrythingy.bind('', - self.print_contents) + self.print_contents) def print_contents(self, event): - print("hi. contents of entry is now ---->", + print("Hi. The current entry content is:", self.contents.get()) + root = tk.Tk() + myapp = App(root) + myapp.mainloop() The Window Manager ^^^^^^^^^^^^^^^^^^ @@ -860,4 +864,4 @@ use raw reads or ``os.read(file.fileno(), maxbytecount)``. WRITABLE EXCEPTION - Constants used in the *mask* arguments. \ No newline at end of file + Constants used in the *mask* arguments. diff --git a/Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst b/Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst new file mode 100644 index 0000000000000..617df72faeb37 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst @@ -0,0 +1 @@ +Make tkinter doc example runnable. From webhook-mailer at python.org Mon Aug 3 00:25:13 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 03 Aug 2020 04:25:13 -0000 Subject: [Python-checkins] bpo-41425: Make tkinter doc example runnable (GH-21706) Message-ID: https://github.com/python/cpython/commit/4bc8445c392e22fb926eeea50d3e943b6241affa commit: 4bc8445c392e22fb926eeea50d3e943b6241affa branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-03T00:25:04-04:00 summary: bpo-41425: Make tkinter doc example runnable (GH-21706) Co-authored-by: Ankit Chandawala Co-authored-by: Terry Jan Reedy (cherry picked from commit c36dbac588e1d99975f285a874bb20e9f5040af4) files: A Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst M Doc/library/tkinter.rst diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index a04926b44612e..e28664d680a04 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -541,31 +541,35 @@ the variable, with no further intervention on your part. For example:: - class App(Frame): - def __init__(self, master=None): + import tkinter as tk + + class App(tk.Frame): + def __init__(self, master): super().__init__(master) self.pack() - self.entrythingy = Entry() + self.entrythingy = tk.Entry() self.entrythingy.pack() - # here is the application variable - self.contents = StringVar() - # set it to some value + # Create the application variable. + self.contents = tk.StringVar() + # Set it to some value. self.contents.set("this is a variable") - # tell the entry widget to watch this variable + # Tell the entry widget to watch this variable. self.entrythingy["textvariable"] = self.contents - # and here we get a callback when the user hits return. - # we will have the program print out the value of the - # application variable when the user hits return + # Define a callback for when the user hits return. + # It prints the current value of the variable. self.entrythingy.bind('', - self.print_contents) + self.print_contents) def print_contents(self, event): - print("hi. contents of entry is now ---->", + print("Hi. The current entry content is:", self.contents.get()) + root = tk.Tk() + myapp = App(root) + myapp.mainloop() The Window Manager ^^^^^^^^^^^^^^^^^^ diff --git a/Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst b/Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst new file mode 100644 index 0000000000000..617df72faeb37 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-03-01-59-48.bpo-41425.KJo6zF.rst @@ -0,0 +1 @@ +Make tkinter doc example runnable. From webhook-mailer at python.org Mon Aug 3 12:04:21 2020 From: webhook-mailer at python.org (Guido van Rossum) Date: Mon, 03 Aug 2020 16:04:21 -0000 Subject: [Python-checkins] A (very) slight speed improvement for iterating over bytes (#21705) Message-ID: https://github.com/python/cpython/commit/488512bf4953d856fcb049a05060a450af52fcdc commit: 488512bf4953d856fcb049a05060a450af52fcdc branch: master author: Guido van Rossum committer: GitHub date: 2020-08-03T09:04:13-07:00 summary: A (very) slight speed improvement for iterating over bytes (#21705) My mentee @xvxvxvxvxv noticed that iterating over array.array is slightly faster than iterating over bytes. Looking at the source I observed that arrayiter_next() calls `getitem(ao, it->index++)` wheras striter_next() uses the idiom (paraphrased) item = PyLong_FromLong(seq->ob_sval[it->it_index]); if (item != NULL) ++it->it_next; return item; I'm not 100% sure but I think that the second version has fewer opportunity for the CPU to overlap the `index++` operation with the rest of the code (which in both cases involves a call). So here I am optimistically incrementing the index -- if the PyLong_FromLong() call fails, this will leave the iterator pointing at the next byte, but honestly I doubt that anyone would seriously consider resuming use of the iterator after that kind of failure (it would have to be a MemoryError). And the author of arrayiter_next() made the same consideration (or never ever gave it a thought :-). With this, a loop like for _ in b: pass is now slightly *faster* than the same thing over an equivalent array, rather than slightly *slower* (in both cases a few percent). files: M Objects/bytesobject.c diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 3a922d32b16e4..836a736037ba4 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -3139,7 +3139,6 @@ static PyObject * striter_next(striterobject *it) { PyBytesObject *seq; - PyObject *item; assert(it != NULL); seq = it->it_seq; @@ -3148,11 +3147,8 @@ striter_next(striterobject *it) assert(PyBytes_Check(seq)); if (it->it_index < PyBytes_GET_SIZE(seq)) { - item = PyLong_FromLong( - (unsigned char)seq->ob_sval[it->it_index]); - if (item != NULL) - ++it->it_index; - return item; + return PyLong_FromLong( + (unsigned char)seq->ob_sval[it->it_index++]); } it->it_seq = NULL; From webhook-mailer at python.org Mon Aug 3 12:41:33 2020 From: webhook-mailer at python.org (Hai Shi) Date: Mon, 03 Aug 2020 16:41:33 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21449) Message-ID: https://github.com/python/cpython/commit/a7f5d93bb6906d0f999248b47295d3a59b130f4d commit: a7f5d93bb6906d0f999248b47295d3a59b130f4d branch: master author: Hai Shi committer: GitHub date: 2020-08-03T18:41:24+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21449) files: M Lib/test/test__opcode.py M Lib/test/test_asyncore.py M Lib/test/test_binascii.py M Lib/test/test_bisect.py M Lib/test/test_builtin.py M Lib/test/test_concurrent_futures.py M Lib/test/test_configparser.py M Lib/test/test_coroutines.py M Lib/test/test_curses.py M Lib/test/test_datetime.py M Lib/test/test_mmap.py M Lib/test/test_signal.py M Lib/test/test_ssl.py M Lib/test/test_tarfile.py M Lib/test/test_threading.py M Lib/test/test_trace.py M Lib/test/test_tracemalloc.py M Lib/test/test_xml_etree_c.py M Lib/unittest/test/test_discovery.py M Lib/unittest/test/test_result.py diff --git a/Lib/test/test__opcode.py b/Lib/test/test__opcode.py index 0fb39eed60642..3bb64a798b3e3 100644 --- a/Lib/test/test__opcode.py +++ b/Lib/test/test__opcode.py @@ -1,5 +1,5 @@ import dis -from test.support import import_module +from test.support.import_helper import import_module import unittest _opcode = import_module("_opcode") diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py index 2cee6fb2e996a..06c6bc2e99dc0 100644 --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -10,8 +10,10 @@ import threading from test import support +from test.support import os_helper from test.support import socket_helper from test.support import threading_helper +from test.support import warnings_helper from io import BytesIO if support.PGO: @@ -92,7 +94,7 @@ def bind_af_aware(sock, addr): """Helper function to bind a socket according to its family.""" if HAS_UNIX_SOCKETS and sock.family == socket.AF_UNIX: # Make sure the path doesn't exist. - support.unlink(addr) + os_helper.unlink(addr) socket_helper.bind_unix_socket(sock, addr) else: sock.bind(addr) @@ -369,14 +371,14 @@ def test_send(self): class FileWrapperTest(unittest.TestCase): def setUp(self): self.d = b"It's not dead, it's sleeping!" - with open(support.TESTFN, 'wb') as file: + with open(os_helper.TESTFN, 'wb') as file: file.write(self.d) def tearDown(self): - support.unlink(support.TESTFN) + os_helper.unlink(os_helper.TESTFN) def test_recv(self): - fd = os.open(support.TESTFN, os.O_RDONLY) + fd = os.open(os_helper.TESTFN, os.O_RDONLY) w = asyncore.file_wrapper(fd) os.close(fd) @@ -390,20 +392,20 @@ def test_recv(self): def test_send(self): d1 = b"Come again?" d2 = b"I want to buy some cheese." - fd = os.open(support.TESTFN, os.O_WRONLY | os.O_APPEND) + fd = os.open(os_helper.TESTFN, os.O_WRONLY | os.O_APPEND) w = asyncore.file_wrapper(fd) os.close(fd) w.write(d1) w.send(d2) w.close() - with open(support.TESTFN, 'rb') as file: + with open(os_helper.TESTFN, 'rb') as file: self.assertEqual(file.read(), self.d + d1 + d2) @unittest.skipUnless(hasattr(asyncore, 'file_dispatcher'), 'asyncore.file_dispatcher required') def test_dispatcher(self): - fd = os.open(support.TESTFN, os.O_RDONLY) + fd = os.open(os_helper.TESTFN, os.O_RDONLY) data = [] class FileDispatcher(asyncore.file_dispatcher): def handle_read(self): @@ -415,16 +417,16 @@ def handle_read(self): def test_resource_warning(self): # Issue #11453 - fd = os.open(support.TESTFN, os.O_RDONLY) + fd = os.open(os_helper.TESTFN, os.O_RDONLY) f = asyncore.file_wrapper(fd) os.close(fd) - with support.check_warnings(('', ResourceWarning)): + with warnings_helper.check_warnings(('', ResourceWarning)): f = None support.gc_collect() def test_close_twice(self): - fd = os.open(support.TESTFN, os.O_RDONLY) + fd = os.open(os_helper.TESTFN, os.O_RDONLY) f = asyncore.file_wrapper(fd) os.close(fd) @@ -804,10 +806,10 @@ class TestAPI_UseIPv6Sockets(BaseTestAPI): class TestAPI_UseUnixSockets(BaseTestAPI): if HAS_UNIX_SOCKETS: family = socket.AF_UNIX - addr = support.TESTFN + addr = os_helper.TESTFN def tearDown(self): - support.unlink(self.addr) + os_helper.unlink(self.addr) BaseTestAPI.tearDown(self) class TestAPI_UseIPv4Select(TestAPI_UseIPv4Sockets, unittest.TestCase): diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py index 45327953a7701..4d1bf2cce1f1e 100644 --- a/Lib/test/test_binascii.py +++ b/Lib/test/test_binascii.py @@ -4,7 +4,8 @@ import binascii import array import re -from test import support +from test.support import warnings_helper + # Note: "*_hex" functions are aliases for "(un)hexlify" b2a_functions = ['b2a_base64', 'b2a_hex', 'b2a_hqx', 'b2a_qp', 'b2a_uu', @@ -37,7 +38,7 @@ def test_functions(self): self.assertTrue(hasattr(getattr(binascii, name), '__call__')) self.assertRaises(TypeError, getattr(binascii, name)) - @support.ignore_warnings(category=DeprecationWarning) + @warnings_helper.ignore_warnings(category=DeprecationWarning) def test_returned_value(self): # Limit to the minimum of all limits (b2a_uu) MAX_ALL = 45 @@ -181,7 +182,7 @@ def test_uu(self): with self.assertRaises(TypeError): binascii.b2a_uu(b"", True) - @support.ignore_warnings(category=DeprecationWarning) + @warnings_helper.ignore_warnings(category=DeprecationWarning) def test_crc_hqx(self): crc = binascii.crc_hqx(self.type2test(b"Test the CRC-32 of"), 0) crc = binascii.crc_hqx(self.type2test(b" this string."), crc) @@ -201,7 +202,7 @@ def test_crc32(self): self.assertRaises(TypeError, binascii.crc32) - @support.ignore_warnings(category=DeprecationWarning) + @warnings_helper.ignore_warnings(category=DeprecationWarning) def test_hqx(self): # Perform binhex4 style RLE-compression # Then calculate the hexbin4 binary-to-ASCII translation @@ -212,7 +213,7 @@ def test_hqx(self): res = binascii.rledecode_hqx(b) self.assertEqual(res, self.rawdata) - @support.ignore_warnings(category=DeprecationWarning) + @warnings_helper.ignore_warnings(category=DeprecationWarning) def test_rle(self): # test repetition with a repetition longer than the limit of 255 data = (b'a' * 100 + b'b' + b'c' * 300) @@ -359,7 +360,7 @@ def test_qp(self): self.assertEqual(b2a_qp(type2test(b'a.\n')), b'a.\n') self.assertEqual(b2a_qp(type2test(b'.a')[:-1]), b'=2E') - @support.ignore_warnings(category=DeprecationWarning) + @warnings_helper.ignore_warnings(category=DeprecationWarning) def test_empty_string(self): # A test for SF bug #1022953. Make sure SystemError is not raised. empty = self.type2test(b'') @@ -384,7 +385,7 @@ def test_unicode_b2a(self): # crc_hqx needs 2 arguments self.assertRaises(TypeError, binascii.crc_hqx, "test", 0) - @support.ignore_warnings(category=DeprecationWarning) + @warnings_helper.ignore_warnings(category=DeprecationWarning) def test_unicode_a2b(self): # Unicode strings are accepted by a2b_* functions. MAX_ALL = 45 diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py index 580a963f627a3..fc7990d765e53 100644 --- a/Lib/test/test_bisect.py +++ b/Lib/test/test_bisect.py @@ -1,10 +1,11 @@ import sys import unittest -from test import support +from test.support import import_helper from collections import UserList -py_bisect = support.import_fresh_module('bisect', blocked=['_bisect']) -c_bisect = support.import_fresh_module('bisect', fresh=['_bisect']) + +py_bisect = import_helper.import_fresh_module('bisect', blocked=['_bisect']) +c_bisect = import_helper.import_fresh_module('bisect', fresh=['_bisect']) class Range(object): """A trivial range()-like object that has an insert() method.""" diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py index 3bc1a3e246fdf..edb4ec092e358 100644 --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -26,10 +26,10 @@ from types import AsyncGeneratorType, FunctionType from operator import neg from test import support -from test.support import ( - EnvironmentVarGuard, TESTFN, check_warnings, swap_attr, unlink, - maybe_get_event_loop_policy) +from test.support import (swap_attr, maybe_get_event_loop_policy) +from test.support.os_helper import (EnvironmentVarGuard, TESTFN, unlink) from test.support.script_helper import assert_python_ok +from test.support.warnings_helper import check_warnings from unittest.mock import MagicMock, patch try: import pty, signal diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index 7da967ea6ced5..a182b14fb9bc0 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -1,8 +1,9 @@ from test import support +from test.support import import_helper from test.support import threading_helper # Skip tests if _multiprocessing wasn't built. -support.import_module('_multiprocessing') +import_helper.import_module('_multiprocessing') # Skip tests if sem_open implementation is broken. support.skip_if_broken_multiprocessing_synchronize() diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index f16da116a745f..230ffc1ccf81a 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -8,6 +8,7 @@ import warnings from test import support +from test.support import os_helper class SortedDict(collections.UserDict): @@ -1063,17 +1064,17 @@ def setUp(self): cf.add_section(s) for j in range(10): cf.set(s, 'lovely_spam{}'.format(j), self.wonderful_spam) - with open(support.TESTFN, 'w') as f: + with open(os_helper.TESTFN, 'w') as f: cf.write(f) def tearDown(self): - os.unlink(support.TESTFN) + os.unlink(os_helper.TESTFN) def test_dominating_multiline_values(self): # We're reading from file because this is where the code changed # during performance updates in Python 3.2 cf_from_file = self.newconfig() - with open(support.TESTFN) as f: + with open(os_helper.TESTFN) as f: cf_from_file.read_file(f) self.assertEqual(cf_from_file.get('section8', 'lovely_spam4'), self.wonderful_spam.replace('\t\n', '\n')) diff --git a/Lib/test/test_coroutines.py b/Lib/test/test_coroutines.py index 8d1e0692a2422..145adb6778170 100644 --- a/Lib/test/test_coroutines.py +++ b/Lib/test/test_coroutines.py @@ -7,6 +7,8 @@ import unittest import warnings from test import support +from test.support import import_helper +from test.support import warnings_helper from test.support.script_helper import assert_python_ok @@ -2117,7 +2119,7 @@ class CoroAsyncIOCompatTest(unittest.TestCase): def test_asyncio_1(self): # asyncio cannot be imported when Python is compiled without thread # support - asyncio = support.import_module('asyncio') + asyncio = import_helper.import_module('asyncio') class MyException(Exception): pass @@ -2258,8 +2260,9 @@ async def corofn(): try: warnings._warn_unawaited_coroutine = lambda coro: 1/0 with support.catch_unraisable_exception() as cm, \ - support.check_warnings((r'coroutine .* was never awaited', - RuntimeWarning)): + warnings_helper.check_warnings( + (r'coroutine .* was never awaited', + RuntimeWarning)): # only store repr() to avoid keeping the coroutine alive coro = corofn() coro_repr = repr(coro) @@ -2272,8 +2275,8 @@ async def corofn(): self.assertEqual(cm.unraisable.exc_type, ZeroDivisionError) del warnings._warn_unawaited_coroutine - with support.check_warnings((r'coroutine .* was never awaited', - RuntimeWarning)): + with warnings_helper.check_warnings( + (r'coroutine .* was never awaited', RuntimeWarning)): corofn() support.gc_collect() diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 5e619d13836d2..2c6d14c3f79dd 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -15,7 +15,8 @@ import tempfile import unittest -from test.support import requires, import_module, verbose, SaveSignals +from test.support import requires, verbose, SaveSignals +from test.support.import_helper import import_module # Optionally test curses module. This currently requires that the # 'curses' resource be given on the regrtest command line using the -u diff --git a/Lib/test/test_datetime.py b/Lib/test/test_datetime.py index d659f369d54e4..bdb9f02e5756a 100644 --- a/Lib/test/test_datetime.py +++ b/Lib/test/test_datetime.py @@ -1,7 +1,9 @@ import unittest import sys -from test.support import import_fresh_module, run_unittest +from test.support import run_unittest +from test.support.import_helper import import_fresh_module + TESTS = 'test.datetimetester' diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 5400f25f50800..8f34c182f82ea 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -1,5 +1,6 @@ -from test.support import (TESTFN, import_module, unlink, - requires, _2G, _4G, gc_collect, cpython_only) +from test.support import (requires, _2G, _4G, gc_collect, cpython_only) +from test.support.import_helper import import_module +from test.support.os_helper import TESTFN, unlink import unittest import os import re diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py index 45553a6a42de7..c6567906321fe 100644 --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -9,6 +9,7 @@ import time import unittest from test import support +from test.support import os_helper from test.support.script_helper import assert_python_ok, spawn_python try: import _testcapi @@ -154,7 +155,7 @@ def test_invalid_call(self): signal.set_wakeup_fd(signal.SIGINT, False) def test_invalid_fd(self): - fd = support.make_bad_fd() + fd = os_helper.make_bad_fd() self.assertRaises((ValueError, OSError), signal.set_wakeup_fd, fd) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index de778d34658b7..26eec969a82e0 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -4,8 +4,11 @@ import unittest import unittest.mock from test import support +from test.support import import_helper +from test.support import os_helper from test.support import socket_helper from test.support import threading_helper +from test.support import warnings_helper import socket import select import time @@ -27,7 +30,7 @@ except ImportError: ctypes = None -ssl = support.import_module("ssl") +ssl = import_helper.import_module("ssl") from ssl import TLSVersion, _TLSContentType, _TLSMessageType @@ -571,7 +574,7 @@ def test_refcycle(self): s = socket.socket(socket.AF_INET) ss = test_wrap_socket(s) wr = weakref.ref(ss) - with support.check_warnings(("", ResourceWarning)): + with warnings_helper.check_warnings(("", ResourceWarning)): del ss self.assertEqual(wr(), None) @@ -893,7 +896,7 @@ def test_get_default_verify_paths(self): self.assertEqual(len(paths), 6) self.assertIsInstance(paths, ssl.DefaultVerifyPaths) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["SSL_CERT_DIR"] = CAPATH env["SSL_CERT_FILE"] = CERTFILE paths = ssl.get_default_verify_paths() @@ -1605,7 +1608,7 @@ def test_load_default_certs(self): @unittest.skipIf(IS_LIBRESSL, "LibreSSL doesn't support env vars") def test_load_default_certs_env(self): ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["SSL_CERT_DIR"] = CAPATH env["SSL_CERT_FILE"] = CERTFILE ctx.load_default_certs() @@ -1619,7 +1622,7 @@ def test_load_default_certs_env_windows(self): stats = ctx.cert_store_stats() ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["SSL_CERT_DIR"] = CAPATH env["SSL_CERT_FILE"] = CERTFILE ctx.load_default_certs() @@ -4266,9 +4269,9 @@ def test_read_write_after_close_raises_valuerror(self): def test_sendfile(self): TEST_DATA = b"x" * 512 - with open(support.TESTFN, 'wb') as f: + with open(os_helper.TESTFN, 'wb') as f: f.write(TEST_DATA) - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) context = ssl.SSLContext(ssl.PROTOCOL_TLS) context.verify_mode = ssl.CERT_REQUIRED context.load_verify_locations(SIGNING_CA) @@ -4277,7 +4280,7 @@ def test_sendfile(self): with server: with context.wrap_socket(socket.socket()) as s: s.connect((HOST, server.port)) - with open(support.TESTFN, 'rb') as file: + with open(os_helper.TESTFN, 'rb') as file: s.sendfile(file) self.assertEqual(s.recv(1024), TEST_DATA) @@ -4603,21 +4606,21 @@ def test_bpo37428_pha_cert_none(self): class TestSSLDebug(unittest.TestCase): - def keylog_lines(self, fname=support.TESTFN): + def keylog_lines(self, fname=os_helper.TESTFN): with open(fname) as f: return len(list(f)) @requires_keylog @unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows") def test_keylog_defaults(self): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) self.assertEqual(ctx.keylog_filename, None) - self.assertFalse(os.path.isfile(support.TESTFN)) - ctx.keylog_filename = support.TESTFN - self.assertEqual(ctx.keylog_filename, support.TESTFN) - self.assertTrue(os.path.isfile(support.TESTFN)) + self.assertFalse(os.path.isfile(os_helper.TESTFN)) + ctx.keylog_filename = os_helper.TESTFN + self.assertEqual(ctx.keylog_filename, os_helper.TESTFN) + self.assertTrue(os.path.isfile(os_helper.TESTFN)) self.assertEqual(self.keylog_lines(), 1) ctx.keylog_filename = None @@ -4626,7 +4629,7 @@ def test_keylog_defaults(self): with self.assertRaises((IsADirectoryError, PermissionError)): # Windows raises PermissionError ctx.keylog_filename = os.path.dirname( - os.path.abspath(support.TESTFN)) + os.path.abspath(os_helper.TESTFN)) with self.assertRaises(TypeError): ctx.keylog_filename = 1 @@ -4634,10 +4637,10 @@ def test_keylog_defaults(self): @requires_keylog @unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows") def test_keylog_filename(self): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) client_context, server_context, hostname = testing_context() - client_context.keylog_filename = support.TESTFN + client_context.keylog_filename = os_helper.TESTFN server = ThreadedEchoServer(context=server_context, chatty=False) with server: with client_context.wrap_socket(socket.socket(), @@ -4647,7 +4650,7 @@ def test_keylog_filename(self): self.assertEqual(self.keylog_lines(), 6) client_context.keylog_filename = None - server_context.keylog_filename = support.TESTFN + server_context.keylog_filename = os_helper.TESTFN server = ThreadedEchoServer(context=server_context, chatty=False) with server: with client_context.wrap_socket(socket.socket(), @@ -4655,8 +4658,8 @@ def test_keylog_filename(self): s.connect((HOST, server.port)) self.assertGreaterEqual(self.keylog_lines(), 11) - client_context.keylog_filename = support.TESTFN - server_context.keylog_filename = support.TESTFN + client_context.keylog_filename = os_helper.TESTFN + server_context.keylog_filename = os_helper.TESTFN server = ThreadedEchoServer(context=server_context, chatty=False) with server: with client_context.wrap_socket(socket.socket(), @@ -4672,19 +4675,19 @@ def test_keylog_filename(self): "test is not compatible with ignore_environment") @unittest.skipIf(Py_DEBUG_WIN32, "Avoid mixing debug/release CRT on Windows") def test_keylog_env(self): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) with unittest.mock.patch.dict(os.environ): - os.environ['SSLKEYLOGFILE'] = support.TESTFN - self.assertEqual(os.environ['SSLKEYLOGFILE'], support.TESTFN) + os.environ['SSLKEYLOGFILE'] = os_helper.TESTFN + self.assertEqual(os.environ['SSLKEYLOGFILE'], os_helper.TESTFN) ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) self.assertEqual(ctx.keylog_filename, None) ctx = ssl.create_default_context() - self.assertEqual(ctx.keylog_filename, support.TESTFN) + self.assertEqual(ctx.keylog_filename, os_helper.TESTFN) ctx = ssl._create_stdlib_context() - self.assertEqual(ctx.keylog_filename, support.TESTFN) + self.assertEqual(ctx.keylog_filename, os_helper.TESTFN) def test_msg_callback(self): client_context, server_context, hostname = testing_context() diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index 3ddeb97f5268f..4bf7248767c4b 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -11,6 +11,7 @@ import tarfile from test import support +from test.support import os_helper from test.support import script_helper # Check for our compression modules. @@ -30,7 +31,7 @@ def sha256sum(data): return sha256(data).hexdigest() -TEMPDIR = os.path.abspath(support.TESTFN) + "-tardir" +TEMPDIR = os.path.abspath(os_helper.TESTFN) + "-tardir" tarextdir = TEMPDIR + '-extract-test' tarname = support.findfile("testtar.tar") gzipname = os.path.join(TEMPDIR, "testtar.tar.gz") @@ -575,21 +576,21 @@ def test_find_members(self): @unittest.skipUnless(hasattr(os, "link"), "Missing hardlink implementation") - @support.skip_unless_symlink + @os_helper.skip_unless_symlink def test_extract_hardlink(self): # Test hardlink extraction (e.g. bug #857297). with tarfile.open(tarname, errorlevel=1, encoding="iso8859-1") as tar: tar.extract("ustar/regtype", TEMPDIR) - self.addCleanup(support.unlink, os.path.join(TEMPDIR, "ustar/regtype")) + self.addCleanup(os_helper.unlink, os.path.join(TEMPDIR, "ustar/regtype")) tar.extract("ustar/lnktype", TEMPDIR) - self.addCleanup(support.unlink, os.path.join(TEMPDIR, "ustar/lnktype")) + self.addCleanup(os_helper.unlink, os.path.join(TEMPDIR, "ustar/lnktype")) with open(os.path.join(TEMPDIR, "ustar/lnktype"), "rb") as f: data = f.read() self.assertEqual(sha256sum(data), sha256_regtype) tar.extract("ustar/symtype", TEMPDIR) - self.addCleanup(support.unlink, os.path.join(TEMPDIR, "ustar/symtype")) + self.addCleanup(os_helper.unlink, os.path.join(TEMPDIR, "ustar/symtype")) with open(os.path.join(TEMPDIR, "ustar/symtype"), "rb") as f: data = f.read() self.assertEqual(sha256sum(data), sha256_regtype) @@ -622,7 +623,7 @@ def format_mtime(mtime): self.assertEqual(tarinfo.mtime, file_mtime, errmsg) finally: tar.close() - support.rmtree(DIR) + os_helper.rmtree(DIR) def test_extract_directory(self): dirtype = "ustar/dirtype" @@ -637,11 +638,11 @@ def test_extract_directory(self): if sys.platform != "win32": self.assertEqual(os.stat(extracted).st_mode & 0o777, 0o755) finally: - support.rmtree(DIR) + os_helper.rmtree(DIR) def test_extractall_pathlike_name(self): DIR = pathlib.Path(TEMPDIR) / "extractall" - with support.temp_dir(DIR), \ + with os_helper.temp_dir(DIR), \ tarfile.open(tarname, encoding="iso8859-1") as tar: directories = [t for t in tar if t.isdir()] tar.extractall(DIR, directories) @@ -652,7 +653,7 @@ def test_extractall_pathlike_name(self): def test_extract_pathlike_name(self): dirtype = "ustar/dirtype" DIR = pathlib.Path(TEMPDIR) / "extractall" - with support.temp_dir(DIR), \ + with os_helper.temp_dir(DIR), \ tarfile.open(tarname, encoding="iso8859-1") as tar: tarinfo = tar.getmember(dirtype) tar.extract(tarinfo, path=DIR) @@ -676,7 +677,7 @@ def test_init_close_fobj(self): else: self.fail("ReadError not raised") finally: - support.unlink(empty) + os_helper.unlink(empty) def test_parallel_iteration(self): # Issue #16601: Restarting iteration over tarfile continued @@ -1026,7 +1027,7 @@ def _fs_supports_holes(): fobj.write(b'x' * 4096) fobj.truncate() s = os.stat(name) - support.unlink(name) + os_helper.unlink(name) return (s.st_blocks * 512 < s.st_size) else: return False @@ -1171,7 +1172,7 @@ def test_directory_size(self): finally: tar.close() finally: - support.rmdir(path) + os_helper.rmdir(path) # mock the following: # os.listdir: so we know that files are in the wrong order @@ -1193,9 +1194,9 @@ def test_ordered_recursion(self): finally: tar.close() finally: - support.unlink(os.path.join(path, "1")) - support.unlink(os.path.join(path, "2")) - support.rmdir(path) + os_helper.unlink(os.path.join(path, "1")) + os_helper.unlink(os.path.join(path, "2")) + os_helper.rmdir(path) def test_gettarinfo_pathlike_name(self): with tarfile.open(tmpname, self.mode) as tar: @@ -1229,10 +1230,10 @@ def test_link_size(self): finally: tar.close() finally: - support.unlink(target) - support.unlink(link) + os_helper.unlink(target) + os_helper.unlink(link) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink def test_symlink_size(self): path = os.path.join(TEMPDIR, "symlink") os.symlink("link_target", path) @@ -1244,7 +1245,7 @@ def test_symlink_size(self): finally: tar.close() finally: - support.unlink(path) + os_helper.unlink(path) def test_add_self(self): # Test for #1257255. @@ -1257,7 +1258,7 @@ def test_add_self(self): self.assertEqual(tar.getnames(), [], "added the archive to itself") - with support.change_cwd(TEMPDIR): + with os_helper.change_cwd(TEMPDIR): tar.add(dstname) self.assertEqual(tar.getnames(), [], "added the archive to itself") @@ -1270,7 +1271,7 @@ def test_filter(self): try: for name in ("foo", "bar", "baz"): name = os.path.join(tempdir, name) - support.create_empty_file(name) + os_helper.create_empty_file(name) def filter(tarinfo): if os.path.basename(tarinfo.name) == "bar": @@ -1298,7 +1299,7 @@ def filter(tarinfo): finally: tar.close() finally: - support.rmtree(tempdir) + os_helper.rmtree(tempdir) # Guarantee that stored pathnames are not modified. Don't # remove ./ or ../ or double slashes. Still make absolute @@ -1309,7 +1310,7 @@ def _test_pathname(self, path, cmp_path=None, dir=False): # and compare the stored name with the original. foo = os.path.join(TEMPDIR, "foo") if not dir: - support.create_empty_file(foo) + os_helper.create_empty_file(foo) else: os.mkdir(foo) @@ -1326,14 +1327,14 @@ def _test_pathname(self, path, cmp_path=None, dir=False): tar.close() if not dir: - support.unlink(foo) + os_helper.unlink(foo) else: - support.rmdir(foo) + os_helper.rmdir(foo) self.assertEqual(t.name, cmp_path or path.replace(os.sep, "/")) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink def test_extractall_symlinks(self): # Test if extractall works properly when tarfile contains symlinks tempdir = os.path.join(TEMPDIR, "testsymlinks") @@ -1356,8 +1357,8 @@ def test_extractall_symlinks(self): except OSError: self.fail("extractall failed with symlinked files") finally: - support.unlink(temparchive) - support.rmtree(tempdir) + os_helper.unlink(temparchive) + os_helper.rmtree(tempdir) def test_pathnames(self): self._test_pathname("foo") @@ -1385,7 +1386,7 @@ def test_abs_pathnames(self): def test_cwd(self): # Test adding the current working directory. - with support.change_cwd(TEMPDIR): + with os_helper.change_cwd(TEMPDIR): tar = tarfile.open(tmpname, self.mode) try: tar.add(".") @@ -1453,7 +1454,7 @@ def test_file_mode(self): # Test for issue #8464: Create files with correct # permissions. if os.path.exists(tmpname): - support.unlink(tmpname) + os_helper.unlink(tmpname) original_umask = os.umask(0o022) try: @@ -1599,7 +1600,7 @@ def test_headers_written_only_for_device_files(self): self.assertEqual(buf_blk[device_headers], b"0000000\0" * 2) self.assertEqual(buf_reg[device_headers], b"\0" * 16) finally: - support.rmtree(tempdir) + os_helper.rmtree(tempdir) class CreateTest(WriteTestBase, unittest.TestCase): @@ -1609,7 +1610,7 @@ class CreateTest(WriteTestBase, unittest.TestCase): file_path = os.path.join(TEMPDIR, "spameggs42") def setUp(self): - support.unlink(tmpname) + os_helper.unlink(tmpname) @classmethod def setUpClass(cls): @@ -1618,7 +1619,7 @@ def setUpClass(cls): @classmethod def tearDownClass(cls): - support.unlink(cls.file_path) + os_helper.unlink(cls.file_path) def test_create(self): with tarfile.open(tmpname, self.mode) as tobj: @@ -1733,8 +1734,8 @@ def setUp(self): def tearDown(self): self.tar.close() - support.unlink(self.foo) - support.unlink(self.bar) + os_helper.unlink(self.foo) + os_helper.unlink(self.bar) def test_add_twice(self): # The same name will be added as a REGTYPE every @@ -2043,7 +2044,7 @@ class AppendTestBase: def setUp(self): self.tarname = tmpname if os.path.exists(self.tarname): - support.unlink(self.tarname) + os_helper.unlink(self.tarname) def _create_testtar(self, mode="w:"): with tarfile.open(tarname, encoding="iso8859-1") as src: @@ -2288,7 +2289,7 @@ def make_simple_tarfile(self, tar_name): files = [support.findfile('tokenize_tests.txt'), support.findfile('tokenize_tests-no-coding-cookie-' 'and-utf8-bom-sig-only.txt')] - self.addCleanup(support.unlink, tar_name) + self.addCleanup(os_helper.unlink, tar_name) with tarfile.open(tar_name, 'w') as tf: for tardata in files: tf.add(tardata, arcname=os.path.basename(tardata)) @@ -2334,7 +2335,7 @@ def test_test_command_invalid_file(self): self.assertEqual(out, b'') self.assertEqual(rc, 1) finally: - support.unlink(tmpname) + os_helper.unlink(tmpname) def test_list_command(self): for tar_name in testtarnames: @@ -2376,7 +2377,7 @@ def test_create_command(self): with tarfile.open(tmpname) as tar: tar.getmembers() finally: - support.unlink(tmpname) + os_helper.unlink(tmpname) def test_create_command_verbose(self): files = [support.findfile('tokenize_tests.txt'), @@ -2390,7 +2391,7 @@ def test_create_command_verbose(self): with tarfile.open(tmpname) as tar: tar.getmembers() finally: - support.unlink(tmpname) + os_helper.unlink(tmpname) def test_create_command_dotless_filename(self): files = [support.findfile('tokenize_tests.txt')] @@ -2400,7 +2401,7 @@ def test_create_command_dotless_filename(self): with tarfile.open(dotlessname) as tar: tar.getmembers() finally: - support.unlink(dotlessname) + os_helper.unlink(dotlessname) def test_create_command_dot_started_filename(self): tar_name = os.path.join(TEMPDIR, ".testtar") @@ -2411,7 +2412,7 @@ def test_create_command_dot_started_filename(self): with tarfile.open(tar_name) as tar: tar.getmembers() finally: - support.unlink(tar_name) + os_helper.unlink(tar_name) def test_create_command_compressed(self): files = [support.findfile('tokenize_tests.txt'), @@ -2426,41 +2427,41 @@ def test_create_command_compressed(self): with filetype.taropen(tar_name) as tar: tar.getmembers() finally: - support.unlink(tar_name) + os_helper.unlink(tar_name) def test_extract_command(self): self.make_simple_tarfile(tmpname) for opt in '-e', '--extract': try: - with support.temp_cwd(tarextdir): + with os_helper.temp_cwd(tarextdir): out = self.tarfilecmd(opt, tmpname) self.assertEqual(out, b'') finally: - support.rmtree(tarextdir) + os_helper.rmtree(tarextdir) def test_extract_command_verbose(self): self.make_simple_tarfile(tmpname) for opt in '-v', '--verbose': try: - with support.temp_cwd(tarextdir): + with os_helper.temp_cwd(tarextdir): out = self.tarfilecmd(opt, '-e', tmpname, PYTHONIOENCODING='utf-8') self.assertIn(b' file is extracted.', out) finally: - support.rmtree(tarextdir) + os_helper.rmtree(tarextdir) def test_extract_command_different_directory(self): self.make_simple_tarfile(tmpname) try: - with support.temp_cwd(tarextdir): + with os_helper.temp_cwd(tarextdir): out = self.tarfilecmd('-e', tmpname, 'spamdir') self.assertEqual(out, b'') finally: - support.rmtree(tarextdir) + os_helper.rmtree(tarextdir) def test_extract_command_invalid_file(self): zipname = support.findfile('zipdir.zip') - with support.temp_cwd(tarextdir): + with os_helper.temp_cwd(tarextdir): rc, out, err = self.tarfilecmd_failure('-e', zipname) self.assertIn(b' is not a tar archive.', err) self.assertEqual(out, b'') @@ -2723,7 +2724,7 @@ def test_keyword_only(self, mock_geteuid): def setUpModule(): - support.unlink(TEMPDIR) + os_helper.unlink(TEMPDIR) os.makedirs(TEMPDIR) global testtarnames @@ -2734,14 +2735,14 @@ def setUpModule(): # Create compressed tarfiles. for c in GzipTest, Bz2Test, LzmaTest: if c.open: - support.unlink(c.tarname) + os_helper.unlink(c.tarname) testtarnames.append(c.tarname) with c.open(c.tarname, "wb") as tar: tar.write(data) def tearDownModule(): if os.path.exists(TEMPDIR): - support.rmtree(TEMPDIR) + os_helper.rmtree(TEMPDIR) if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index ad82e304e32f3..47e131ae27a14 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -4,7 +4,8 @@ import test.support from test.support import threading_helper -from test.support import verbose, import_module, cpython_only +from test.support import verbose, cpython_only +from test.support.import_helper import import_module from test.support.script_helper import assert_python_ok, assert_python_failure import random diff --git a/Lib/test/test_trace.py b/Lib/test/test_trace.py index c03982ba72b3f..75478557e7871 100644 --- a/Lib/test/test_trace.py +++ b/Lib/test/test_trace.py @@ -1,6 +1,7 @@ import os import sys -from test.support import TESTFN, TESTFN_UNICODE, FS_NONASCII, rmtree, unlink, captured_stdout +from test.support import captured_stdout +from test.support.os_helper import (TESTFN, rmtree, unlink) from test.support.script_helper import assert_python_ok, assert_python_failure import textwrap import unittest diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py index c5ae4e6d653bf..a0037f81b88f2 100644 --- a/Lib/test/test_tracemalloc.py +++ b/Lib/test/test_tracemalloc.py @@ -7,6 +7,7 @@ from test.support.script_helper import (assert_python_ok, assert_python_failure, interpreter_requires_environment) from test import support +from test.support import os_helper try: import _testcapi @@ -287,11 +288,11 @@ def test_snapshot(self): self.assertGreater(snapshot.traces[1].traceback.total_nframe, 10) # write on disk - snapshot.dump(support.TESTFN) - self.addCleanup(support.unlink, support.TESTFN) + snapshot.dump(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) # load from disk - snapshot2 = tracemalloc.Snapshot.load(support.TESTFN) + snapshot2 = tracemalloc.Snapshot.load(os_helper.TESTFN) self.assertEqual(snapshot2.traces, snapshot.traces) # tracemalloc must be tracing memory allocations to take a snapshot @@ -306,11 +307,11 @@ def test_snapshot_save_attr(self): # take a snapshot with a new attribute snapshot = tracemalloc.take_snapshot() snapshot.test_attr = "new" - snapshot.dump(support.TESTFN) - self.addCleanup(support.unlink, support.TESTFN) + snapshot.dump(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) # load() should recreate the attribute - snapshot2 = tracemalloc.Snapshot.load(support.TESTFN) + snapshot2 = tracemalloc.Snapshot.load(os_helper.TESTFN) self.assertEqual(snapshot2.test_attr, "new") def fork_child(self): diff --git a/Lib/test/test_xml_etree_c.py b/Lib/test/test_xml_etree_c.py index e26e1714a540b..e68613bb910dd 100644 --- a/Lib/test/test_xml_etree_c.py +++ b/Lib/test/test_xml_etree_c.py @@ -2,7 +2,7 @@ import io import struct from test import support -from test.support import import_fresh_module +from test.support.import_helper import import_fresh_module import types import unittest diff --git a/Lib/unittest/test/test_discovery.py b/Lib/unittest/test/test_discovery.py index 16e081e1fb76e..9d502c51fb36a 100644 --- a/Lib/unittest/test/test_discovery.py +++ b/Lib/unittest/test/test_discovery.py @@ -5,6 +5,7 @@ import types import pickle from test import support +from test.support import import_helper import test.test_importlib.util import unittest @@ -848,7 +849,7 @@ def _find_tests(start_dir, pattern, namespace=None): with unittest.mock.patch('builtins.__import__', _import): # Since loader.discover() can modify sys.path, restore it when done. - with support.DirsOnSysPath(): + with import_helper.DirsOnSysPath(): # Make sure to remove 'package' from sys.modules when done. with test.test_importlib.util.uncache('package'): suite = loader.discover('package') @@ -865,7 +866,7 @@ def _import(packagename, *args, **kwargs): with unittest.mock.patch('builtins.__import__', _import): # Since loader.discover() can modify sys.path, restore it when done. - with support.DirsOnSysPath(): + with import_helper.DirsOnSysPath(): # Make sure to remove 'package' from sys.modules when done. with test.test_importlib.util.uncache('package'): with self.assertRaises(TypeError) as cm: diff --git a/Lib/unittest/test/test_result.py b/Lib/unittest/test/test_result.py index 0ffb87b40256c..a4af67bd8d56d 100644 --- a/Lib/unittest/test/test_result.py +++ b/Lib/unittest/test/test_result.py @@ -2,7 +2,7 @@ import sys import textwrap -from test import support +from test.support import warnings_helper import traceback import unittest @@ -458,8 +458,8 @@ def __init__(self, stream=None, descriptions=None, verbosity=None): class Test_OldTestResult(unittest.TestCase): def assertOldResultWarning(self, test, failures): - with support.check_warnings(("TestResult has no add.+ method,", - RuntimeWarning)): + with warnings_helper.check_warnings( + ("TestResult has no add.+ method,", RuntimeWarning)): result = OldResult() test.run(result) self.assertEqual(len(result.failures), failures) From webhook-mailer at python.org Mon Aug 3 12:47:46 2020 From: webhook-mailer at python.org (Hai Shi) Date: Mon, 03 Aug 2020 16:47:46 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21451) Message-ID: https://github.com/python/cpython/commit/bb0424b122e3d222a558bd4177ce37befd3e0347 commit: bb0424b122e3d222a558bd4177ce37befd3e0347 branch: master author: Hai Shi committer: GitHub date: 2020-08-03T18:47:42+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21451) files: M Lib/test/test_bdb.py M Lib/test/test_code_module.py M Lib/test/test_embed.py M Lib/test/test_gzip.py M Lib/test/test_largefile.py M Lib/test/test_lltrace.py M Lib/test/test_ordered_dict.py M Lib/test/test_pathlib.py M Lib/test/test_pipes.py M Lib/test/test_posix.py M Lib/test/test_pydoc.py M Lib/test/test_resource.py M Lib/test/test_runpy.py M Lib/test/test_sax.py M Lib/test/test_sys.py M Lib/test/test_syslog.py M Lib/test/test_ttk_textonly.py M Lib/test/test_unicode_file_functions.py M Lib/test/test_urllib2.py M Lib/test/test_venv.py diff --git a/Lib/test/test_bdb.py b/Lib/test/test_bdb.py index ae16880567882..9bce780e0d041 100644 --- a/Lib/test/test_bdb.py +++ b/Lib/test/test_bdb.py @@ -58,6 +58,9 @@ from contextlib import contextmanager from itertools import islice, repeat import test.support +from test.support import import_helper +from test.support import os_helper + class BdbException(Exception): pass class BdbError(BdbException): """Error raised by the Bdb instance.""" @@ -531,7 +534,7 @@ def gen(a, b): @contextmanager def create_modules(modules): - with test.support.temp_cwd(): + with os_helper.temp_cwd(): sys.path.append(os.getcwd()) try: for m in modules: @@ -543,7 +546,7 @@ def create_modules(modules): yield finally: for m in modules: - test.support.forget(m) + import_helper.forget(m) sys.path.pop() def break_in_func(funcname, fname=__file__, temporary=False, cond=None): diff --git a/Lib/test/test_code_module.py b/Lib/test/test_code_module.py index 24db0ace80122..226bc3a853b7b 100644 --- a/Lib/test/test_code_module.py +++ b/Lib/test/test_code_module.py @@ -4,9 +4,10 @@ from textwrap import dedent from contextlib import ExitStack from unittest import mock -from test import support +from test.support import import_helper -code = support.import_module('code') + +code = import_helper.import_module('code') class TestInteractiveConsole(unittest.TestCase): diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 2b740521e723c..31dc39fd9e8ef 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -1,5 +1,7 @@ # Run the tests in Programs/_testembed.c (tests for the CPython embedding APIs) from test import support +from test.support import import_helper +from test.support import os_helper import unittest from collections import namedtuple @@ -1339,8 +1341,8 @@ def test_global_pathconfig(self): # # The global path configuration (_Py_path_config) must be a copy # of the path configuration of PyInterpreter.config (PyConfig). - ctypes = support.import_module('ctypes') - _testinternalcapi = support.import_module('_testinternalcapi') + ctypes = import_helper.import_module('ctypes') + _testinternalcapi = import_helper.import_module('_testinternalcapi') def get_func(name): func = getattr(ctypes.pythonapi, name) @@ -1418,7 +1420,7 @@ def test_audit_run_file(self): returncode=1) def test_audit_run_interactivehook(self): - startup = os.path.join(self.oldcwd, support.TESTFN) + ".py" + startup = os.path.join(self.oldcwd, os_helper.TESTFN) + ".py" with open(startup, "w", encoding="utf-8") as f: print("import sys", file=f) print("sys.__interactivehook__ = lambda: None", file=f) @@ -1431,7 +1433,7 @@ def test_audit_run_interactivehook(self): os.unlink(startup) def test_audit_run_startup(self): - startup = os.path.join(self.oldcwd, support.TESTFN) + ".py" + startup = os.path.join(self.oldcwd, os_helper.TESTFN) + ".py" with open(startup, "w", encoding="utf-8") as f: print("pass", file=f) try: diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py index 0f235d1805e0d..c3fa9b097f8aa 100644 --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -11,10 +11,12 @@ import unittest from subprocess import PIPE, Popen from test import support +from test.support import import_helper +from test.support import os_helper from test.support import _4G, bigmemtest from test.support.script_helper import assert_python_ok, assert_python_failure -gzip = support.import_module('gzip') +gzip = import_helper.import_module('gzip') data1 = b""" int length=DEFAULTALLOC, err = Z_OK; PyObject *RetVal; @@ -29,7 +31,7 @@ """ -TEMPDIR = os.path.abspath(support.TESTFN) + '-gzdir' +TEMPDIR = os.path.abspath(os_helper.TESTFN) + '-gzdir' class UnseekableIO(io.BytesIO): @@ -44,13 +46,13 @@ def seek(self, *args): class BaseTest(unittest.TestCase): - filename = support.TESTFN + filename = os_helper.TESTFN def setUp(self): - support.unlink(self.filename) + os_helper.unlink(self.filename) def tearDown(self): - support.unlink(self.filename) + os_helper.unlink(self.filename) class TestGzip(BaseTest): @@ -286,7 +288,7 @@ def test_mode(self): self.test_write() with gzip.GzipFile(self.filename, 'r') as f: self.assertEqual(f.myfileobj.mode, 'rb') - support.unlink(self.filename) + os_helper.unlink(self.filename) with gzip.GzipFile(self.filename, 'x') as f: self.assertEqual(f.myfileobj.mode, 'xb') @@ -365,7 +367,7 @@ def test_metadata(self): self.assertEqual(isizeBytes, struct.pack(' https://github.com/python/cpython/commit/4660597b51b3d14ce6269d0ed865ab7e22c6ae1f commit: 4660597b51b3d14ce6269d0ed865ab7e22c6ae1f branch: master author: Hai Shi committer: GitHub date: 2020-08-03T18:49:18+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21448) files: M Lib/test/test_argparse.py M Lib/test/test_asyncgen.py M Lib/test/test_clinic.py M Lib/test/test_codecs.py M Lib/test/test_doctest.py M Lib/test/test_exceptions.py M Lib/test/test_ftplib.py M Lib/test/test_gc.py M Lib/test/test_genericpath.py M Lib/test/test_imghdr.py M Lib/test/test_mimetypes.py M Lib/test/test_ntpath.py M Lib/test/test_operator.py M Lib/test/test_optparse.py M Lib/test/test_peg_generator/test_c_parser.py M Lib/test/test_queue.py M Lib/test/test_spwd.py M Lib/test/test_stat.py M Lib/test/test_tokenize.py M Lib/test/test_webbrowser.py diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 22cae626ccc29..e98c15b11afb3 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -13,6 +13,7 @@ from io import StringIO from test import support +from test.support import os_helper from unittest import mock class StdIOBuffer(StringIO): pass @@ -23,7 +24,7 @@ def setUp(self): # The tests assume that line wrapping occurs at 80 columns, but this # behaviour can be overridden by setting the COLUMNS environment # variable. To ensure that this width is used, set COLUMNS to 80. - env = support.EnvironmentVarGuard() + env = os_helper.EnvironmentVarGuard() env['COLUMNS'] = '80' self.addCleanup(env.__exit__) @@ -3244,7 +3245,7 @@ class TestShortColumns(HelpTestCase): but we don't want any exceptions thrown in such cases. Only ugly representation. ''' def setUp(self): - env = support.EnvironmentVarGuard() + env = os_helper.EnvironmentVarGuard() env.set("COLUMNS", '15') self.addCleanup(env.__exit__) diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index 62bf877416652..1f7e05b42be99 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -2,7 +2,7 @@ import types import unittest -from test.support import import_module +from test.support.import_helper import import_module asyncio = import_module("asyncio") diff --git a/Lib/test/test_clinic.py b/Lib/test/test_clinic.py index 3d5dc4759d501..80b9aec7c2f5b 100644 --- a/Lib/test/test_clinic.py +++ b/Lib/test/test_clinic.py @@ -3,6 +3,7 @@ # Licensed to the PSF under a contributor agreement. from test import support, test_tools +from test.support import os_helper from unittest import TestCase import collections import inspect @@ -797,7 +798,7 @@ def test_external(self): source = support.findfile('clinic.test') with open(source, 'r', encoding='utf-8') as f: original = f.read() - with support.temp_dir() as testdir: + with os_helper.temp_dir() as testdir: testfile = os.path.join(testdir, 'clinic.test.c') with open(testfile, 'w', encoding='utf-8') as f: f.write(original) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 54a3520802a4f..f0da35c039e11 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -8,6 +8,8 @@ from unittest import mock from test import support +from test.support import os_helper +from test.support import warnings_helper try: import _testcapi @@ -709,11 +711,11 @@ def test_bug691291(self): s1 = 'Hello\r\nworld\r\n' s = s1.encode(self.encoding) - self.addCleanup(support.unlink, support.TESTFN) - with open(support.TESTFN, 'wb') as fp: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + with open(os_helper.TESTFN, 'wb') as fp: fp.write(s) - with support.check_warnings(('', DeprecationWarning)): - reader = codecs.open(support.TESTFN, 'U', encoding=self.encoding) + with warnings_helper.check_warnings(('', DeprecationWarning)): + reader = codecs.open(os_helper.TESTFN, 'U', encoding=self.encoding) with reader: self.assertEqual(reader.read(), s1) @@ -1697,10 +1699,10 @@ def test_all(self): getattr(codecs, api) def test_open(self): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) for mode in ('w', 'r', 'r+', 'w+', 'a', 'a+'): with self.subTest(mode), \ - codecs.open(support.TESTFN, mode, 'ascii') as file: + codecs.open(os_helper.TESTFN, mode, 'ascii') as file: self.assertIsInstance(file, codecs.StreamReaderWriter) def test_undefined(self): @@ -1718,7 +1720,7 @@ def test_file_closes_if_lookup_error_raised(self): mock_open = mock.mock_open() with mock.patch('builtins.open', mock_open) as file: with self.assertRaises(LookupError): - codecs.open(support.TESTFN, 'wt', 'invalid-encoding') + codecs.open(os_helper.TESTFN, 'wt', 'invalid-encoding') file().close.assert_called() @@ -2516,10 +2518,10 @@ def test_seek0(self): "utf-32", "utf-32-le", "utf-32-be") - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) for encoding in tests: # Check if the BOM is written only once - with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f: + with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.write(data) f.write(data) f.seek(0) @@ -2528,7 +2530,7 @@ def test_seek0(self): self.assertEqual(f.read(), data * 2) # Check that the BOM is written after a seek(0) - with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f: + with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.write(data[0]) self.assertNotEqual(f.tell(), 0) f.seek(0) @@ -2537,7 +2539,7 @@ def test_seek0(self): self.assertEqual(f.read(), data) # (StreamWriter) Check that the BOM is written after a seek(0) - with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f: + with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.writer.write(data[0]) self.assertNotEqual(f.writer.tell(), 0) f.writer.seek(0) @@ -2547,7 +2549,7 @@ def test_seek0(self): # Check that the BOM is not written after a seek() at a position # different than the start - with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f: + with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.write(data) f.seek(f.tell()) f.write(data) @@ -2556,7 +2558,7 @@ def test_seek0(self): # (StreamWriter) Check that the BOM is not written after a seek() # at a position different than the start - with codecs.open(support.TESTFN, 'w+', encoding=encoding) as f: + with codecs.open(os_helper.TESTFN, 'w+', encoding=encoding) as f: f.writer.write(data) f.writer.seek(f.writer.tell()) f.writer.write(data) diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 8d9f872968775..bff20f9cac9c9 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -3,6 +3,8 @@ """ from test import support +from test.support import import_helper +from test.support import os_helper import doctest import functools import os @@ -441,7 +443,7 @@ def basics(): r""" >>> tests = finder.find(sample_func) >>> print(tests) # doctest: +ELLIPSIS - [] + [] The exact name depends on how test_doctest was invoked, so allow for leading path components. @@ -705,7 +707,7 @@ def test_empty_namespace_package(self): try: mod = importlib.import_module(pkg_name) finally: - support.forget(pkg_name) + import_helper.forget(pkg_name) sys.path.pop() include_empty_finder = doctest.DocTestFinder(exclude_empty=False) @@ -2758,7 +2760,7 @@ def test_lineendings(): r""" >>> dn = tempfile.mkdtemp() >>> pkg = os.path.join(dn, "doctest_testpkg") >>> os.mkdir(pkg) - >>> support.create_empty_file(os.path.join(pkg, "__init__.py")) + >>> os_helper.create_empty_file(os.path.join(pkg, "__init__.py")) >>> fn = os.path.join(pkg, "doctest_testfile.txt") >>> with open(fn, 'wb') as f: ... f.write( @@ -2840,7 +2842,8 @@ def test_CLI(): r""" simple tests and no errors. We'll run both the unadorned doctest command, and the verbose version, and then check the output: - >>> from test.support import script_helper, temp_dir + >>> from test.support import script_helper + >>> from test.support.os_helper import temp_dir >>> with temp_dir() as tmpdir: ... fn = os.path.join(tmpdir, 'myfile.doc') ... with open(fn, 'w') as f: @@ -2891,7 +2894,8 @@ def test_CLI(): r""" file ends in '.py', its handling of python module files (as opposed to straight text files). - >>> from test.support import script_helper, temp_dir + >>> from test.support import script_helper + >>> from test.support.os_helper import temp_dir >>> with temp_dir() as tmpdir: ... fn = os.path.join(tmpdir, 'myfile.doc') ... with open(fn, 'w') as f: @@ -3109,7 +3113,7 @@ def test_main(): def test_coverage(coverdir): - trace = support.import_module('trace') + trace = import_helper.import_module('trace') tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix,], trace=0, count=1) tracer.run('test_main()') diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index a67e69bfff728..2ffe8caa03f81 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -8,10 +8,13 @@ import weakref import errno -from test.support import (TESTFN, captured_stderr, check_impl_detail, - check_warnings, cpython_only, gc_collect, - no_tracing, unlink, import_module, script_helper, +from test.support import (captured_stderr, check_impl_detail, + cpython_only, gc_collect, + no_tracing, script_helper, SuppressCrashReport) +from test.support.import_helper import import_module +from test.support.os_helper import TESTFN, unlink +from test.support.warnings_helper import check_warnings from test import support diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py index cb43573318b6a..65feb3aadedd6 100644 --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -21,6 +21,7 @@ from test import support from test.support import threading_helper from test.support import socket_helper +from test.support import warnings_helper from test.support.socket_helper import HOST, HOSTv6 TIMEOUT = support.LOOPBACK_TIMEOUT @@ -623,7 +624,7 @@ def test_storlines(self): f = io.StringIO(RETR_DATA.replace('\r\n', '\n')) # storlines() expects a binary file, not a text file - with support.check_warnings(('', BytesWarning), quiet=True): + with warnings_helper.check_warnings(('', BytesWarning), quiet=True): self.assertRaises(TypeError, self.client.storlines, 'stor foo', f) def test_nlst(self): diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py index c82970827c672..1b096efdbcf5f 100644 --- a/Lib/test/test_gc.py +++ b/Lib/test/test_gc.py @@ -1,8 +1,9 @@ import unittest import unittest.mock from test.support import (verbose, refcount_test, run_unittest, - cpython_only, temp_dir, TESTFN, unlink, - import_module) + cpython_only) +from test.support.import_helper import import_module +from test.support.os_helper import temp_dir, TESTFN, unlink from test.support.script_helper import assert_python_ok, make_script from test.support import threading_helper diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py index e7acbcd29088b..1ff7f75ad3e61 100644 --- a/Lib/test/test_genericpath.py +++ b/Lib/test/test_genericpath.py @@ -7,9 +7,10 @@ import sys import unittest import warnings -from test import support +from test.support import os_helper +from test.support import warnings_helper from test.support.script_helper import assert_python_ok -from test.support import FakePath +from test.support.os_helper import FakePath def create_file(filename, data=b'foo'): @@ -97,8 +98,8 @@ def test_commonprefix(self): self.assertNotEqual(s1[n:n+1], s2[n:n+1]) def test_getsize(self): - filename = support.TESTFN - self.addCleanup(support.unlink, filename) + filename = os_helper.TESTFN + self.addCleanup(os_helper.unlink, filename) create_file(filename, b'Hello') self.assertEqual(self.pathmodule.getsize(filename), 5) @@ -108,8 +109,8 @@ def test_getsize(self): self.assertEqual(self.pathmodule.getsize(filename), 12) def test_filetime(self): - filename = support.TESTFN - self.addCleanup(support.unlink, filename) + filename = os_helper.TESTFN + self.addCleanup(os_helper.unlink, filename) create_file(filename, b'foo') @@ -126,9 +127,9 @@ def test_filetime(self): ) def test_exists(self): - filename = support.TESTFN + filename = os_helper.TESTFN bfilename = os.fsencode(filename) - self.addCleanup(support.unlink, filename) + self.addCleanup(os_helper.unlink, filename) self.assertIs(self.pathmodule.exists(filename), False) self.assertIs(self.pathmodule.exists(bfilename), False) @@ -163,7 +164,7 @@ def test_exists_fd(self): self.assertFalse(self.pathmodule.exists(r)) def test_isdir(self): - filename = support.TESTFN + filename = os_helper.TESTFN bfilename = os.fsencode(filename) self.assertIs(self.pathmodule.isdir(filename), False) self.assertIs(self.pathmodule.isdir(bfilename), False) @@ -178,17 +179,17 @@ def test_isdir(self): self.assertIs(self.pathmodule.isdir(filename), False) self.assertIs(self.pathmodule.isdir(bfilename), False) finally: - support.unlink(filename) + os_helper.unlink(filename) try: os.mkdir(filename) self.assertIs(self.pathmodule.isdir(filename), True) self.assertIs(self.pathmodule.isdir(bfilename), True) finally: - support.rmdir(filename) + os_helper.rmdir(filename) def test_isfile(self): - filename = support.TESTFN + filename = os_helper.TESTFN bfilename = os.fsencode(filename) self.assertIs(self.pathmodule.isfile(filename), False) self.assertIs(self.pathmodule.isfile(bfilename), False) @@ -203,20 +204,20 @@ def test_isfile(self): self.assertIs(self.pathmodule.isfile(filename), True) self.assertIs(self.pathmodule.isfile(bfilename), True) finally: - support.unlink(filename) + os_helper.unlink(filename) try: os.mkdir(filename) self.assertIs(self.pathmodule.isfile(filename), False) self.assertIs(self.pathmodule.isfile(bfilename), False) finally: - support.rmdir(filename) + os_helper.rmdir(filename) def test_samefile(self): - file1 = support.TESTFN - file2 = support.TESTFN + "2" - self.addCleanup(support.unlink, file1) - self.addCleanup(support.unlink, file2) + file1 = os_helper.TESTFN + file2 = os_helper.TESTFN + "2" + self.addCleanup(os_helper.unlink, file1) + self.addCleanup(os_helper.unlink, file2) create_file(file1) self.assertTrue(self.pathmodule.samefile(file1, file1)) @@ -227,10 +228,10 @@ def test_samefile(self): self.assertRaises(TypeError, self.pathmodule.samefile) def _test_samefile_on_link_func(self, func): - test_fn1 = support.TESTFN - test_fn2 = support.TESTFN + "2" - self.addCleanup(support.unlink, test_fn1) - self.addCleanup(support.unlink, test_fn2) + test_fn1 = os_helper.TESTFN + test_fn2 = os_helper.TESTFN + "2" + self.addCleanup(os_helper.unlink, test_fn1) + self.addCleanup(os_helper.unlink, test_fn2) create_file(test_fn1) @@ -241,7 +242,7 @@ def _test_samefile_on_link_func(self, func): create_file(test_fn2) self.assertFalse(self.pathmodule.samefile(test_fn1, test_fn2)) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink def test_samefile_on_symlink(self): self._test_samefile_on_link_func(os.symlink) @@ -252,10 +253,10 @@ def test_samefile_on_link(self): self.skipTest('os.link(): %s' % e) def test_samestat(self): - test_fn1 = support.TESTFN - test_fn2 = support.TESTFN + "2" - self.addCleanup(support.unlink, test_fn1) - self.addCleanup(support.unlink, test_fn2) + test_fn1 = os_helper.TESTFN + test_fn2 = os_helper.TESTFN + "2" + self.addCleanup(os_helper.unlink, test_fn1) + self.addCleanup(os_helper.unlink, test_fn2) create_file(test_fn1) stat1 = os.stat(test_fn1) @@ -268,10 +269,10 @@ def test_samestat(self): self.assertRaises(TypeError, self.pathmodule.samestat) def _test_samestat_on_link_func(self, func): - test_fn1 = support.TESTFN + "1" - test_fn2 = support.TESTFN + "2" - self.addCleanup(support.unlink, test_fn1) - self.addCleanup(support.unlink, test_fn2) + test_fn1 = os_helper.TESTFN + "1" + test_fn2 = os_helper.TESTFN + "2" + self.addCleanup(os_helper.unlink, test_fn1) + self.addCleanup(os_helper.unlink, test_fn2) create_file(test_fn1) func(test_fn1, test_fn2) @@ -283,7 +284,7 @@ def _test_samestat_on_link_func(self, func): self.assertFalse(self.pathmodule.samestat(os.stat(test_fn1), os.stat(test_fn2))) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink def test_samestat_on_symlink(self): self._test_samestat_on_link_func(os.symlink) @@ -294,8 +295,8 @@ def test_samestat_on_link(self): self.skipTest('os.link(): %s' % e) def test_sameopenfile(self): - filename = support.TESTFN - self.addCleanup(support.unlink, filename) + filename = os_helper.TESTFN + self.addCleanup(os_helper.unlink, filename) create_file(filename) with open(filename, "rb", 0) as fp1: @@ -374,7 +375,7 @@ def test_splitdrive(self): def test_expandvars(self): expandvars = self.pathmodule.expandvars - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.clear() env["foo"] = "bar" env["{foo"] = "baz1" @@ -403,14 +404,14 @@ def test_expandvars(self): self.assertEqual(expandvars(b"$foo$foo"), b"barbar") self.assertEqual(expandvars(b"$bar$bar"), b"$bar$bar") - @unittest.skipUnless(support.FS_NONASCII, 'need support.FS_NONASCII') + @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') def test_expandvars_nonascii(self): expandvars = self.pathmodule.expandvars def check(value, expected): self.assertEqual(expandvars(value), expected) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.clear() - nonascii = support.FS_NONASCII + nonascii = os_helper.FS_NONASCII env['spam'] = nonascii env[nonascii] = 'ham' + nonascii check(nonascii, nonascii) @@ -469,31 +470,31 @@ def test_abspath_issue3426(self): # FS encoding is probably ASCII pass else: - with support.temp_cwd(unicwd): + with os_helper.temp_cwd(unicwd): for path in ('', 'fuu', 'f\xf9\xf9', '/fuu', 'U:\\'): self.assertIsInstance(abspath(path), str) def test_nonascii_abspath(self): - if (support.TESTFN_UNDECODABLE + if (os_helper.TESTFN_UNDECODABLE # Mac OS X denies the creation of a directory with an invalid # UTF-8 name. Windows allows creating a directory with an # arbitrary bytes name, but fails to enter this directory # (when the bytes name is used). and sys.platform not in ('win32', 'darwin')): - name = support.TESTFN_UNDECODABLE - elif support.TESTFN_NONASCII: - name = support.TESTFN_NONASCII + name = os_helper.TESTFN_UNDECODABLE + elif os_helper.TESTFN_NONASCII: + name = os_helper.TESTFN_NONASCII else: - self.skipTest("need support.TESTFN_NONASCII") + self.skipTest("need os_helper.TESTFN_NONASCII") with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) - with support.temp_cwd(name): + with os_helper.temp_cwd(name): self.test_abspath() def test_join_errors(self): # Check join() raises friendly TypeErrors. - with support.check_warnings(('', BytesWarning), quiet=True): + with warnings_helper.check_warnings(('', BytesWarning), quiet=True): errmsg = "Can't mix strings and bytes in path components" with self.assertRaisesRegex(TypeError, errmsg): self.pathmodule.join(b'bytes', 'str') @@ -513,8 +514,8 @@ def test_join_errors(self): def test_relpath_errors(self): # Check relpath() raises friendly TypeErrors. - with support.check_warnings(('', (BytesWarning, DeprecationWarning)), - quiet=True): + with warnings_helper.check_warnings( + ('', (BytesWarning, DeprecationWarning)), quiet=True): errmsg = "Can't mix strings and bytes in path components" with self.assertRaisesRegex(TypeError, errmsg): self.pathmodule.relpath(b'bytes', 'str') @@ -534,9 +535,9 @@ def test_import(self): class PathLikeTests(unittest.TestCase): def setUp(self): - self.file_name = support.TESTFN - self.file_path = FakePath(support.TESTFN) - self.addCleanup(support.unlink, self.file_name) + self.file_name = os_helper.TESTFN + self.file_path = FakePath(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, self.file_name) create_file(self.file_name, b"test_genericpath.PathLikeTests") def assertPathEqual(self, func): diff --git a/Lib/test/test_imghdr.py b/Lib/test/test_imghdr.py index 476ba95f173c1..b2d1fc8322a03 100644 --- a/Lib/test/test_imghdr.py +++ b/Lib/test/test_imghdr.py @@ -4,7 +4,9 @@ import pathlib import unittest import warnings -from test.support import findfile, TESTFN, unlink +from test.support import findfile +from test.support.os_helper import TESTFN, unlink + TEST_FILES = ( ('python.png', 'png'), diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py index 683d393fdb491..ddeae38e1372f 100644 --- a/Lib/test/test_mimetypes.py +++ b/Lib/test/test_mimetypes.py @@ -6,6 +6,7 @@ import unittest from test import support +from test.support import os_helper from platform import win32_edition @@ -60,7 +61,7 @@ def test_read_mime_types(self): # Unreadable file returns None self.assertIsNone(mimetypes.read_mime_types("non-existent")) - with support.temp_dir() as directory: + with os_helper.temp_dir() as directory: data = "x-application/x-unittest pyunit\n" file = pathlib.Path(directory, "sample.mimetype") file.write_text(data) @@ -70,7 +71,7 @@ def test_read_mime_types(self): # bpo-41048: read_mime_types should read the rule file with 'utf-8' encoding. # Not with locale encoding. _bootlocale has been imported because io.open(...) # uses it. - with support.temp_dir() as directory: + with os_helper.temp_dir() as directory: data = "application/no-mans-land Fran\u00E7ais" file = pathlib.Path(directory, "sample.mimetype") file.write_text(data, encoding='utf-8') diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py index 6f881f197c4f5..69c44710f0b5a 100644 --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -3,7 +3,9 @@ import sys import unittest import warnings -from test.support import TestFailed, FakePath +from test.support import os_helper +from test.support import TestFailed +from test.support.os_helper import FakePath from test import support, test_genericpath from tempfile import TemporaryFile @@ -254,34 +256,34 @@ def test_realpath_pardir(self): tester("ntpath.realpath('\\'.join(['..'] * 50))", ntpath.splitdrive(expected)[0] + '\\') - @support.skip_unless_symlink + @os_helper.skip_unless_symlink @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname') def test_realpath_basic(self): - ABSTFN = ntpath.abspath(support.TESTFN) + ABSTFN = ntpath.abspath(os_helper.TESTFN) open(ABSTFN, "wb").close() - self.addCleanup(support.unlink, ABSTFN) - self.addCleanup(support.unlink, ABSTFN + "1") + self.addCleanup(os_helper.unlink, ABSTFN) + self.addCleanup(os_helper.unlink, ABSTFN + "1") os.symlink(ABSTFN, ABSTFN + "1") self.assertPathEqual(ntpath.realpath(ABSTFN + "1"), ABSTFN) self.assertPathEqual(ntpath.realpath(os.fsencode(ABSTFN + "1")), os.fsencode(ABSTFN)) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname') def test_realpath_relative(self): - ABSTFN = ntpath.abspath(support.TESTFN) + ABSTFN = ntpath.abspath(os_helper.TESTFN) open(ABSTFN, "wb").close() - self.addCleanup(support.unlink, ABSTFN) - self.addCleanup(support.unlink, ABSTFN + "1") + self.addCleanup(os_helper.unlink, ABSTFN) + self.addCleanup(os_helper.unlink, ABSTFN + "1") os.symlink(ABSTFN, ntpath.relpath(ABSTFN + "1")) self.assertPathEqual(ntpath.realpath(ABSTFN + "1"), ABSTFN) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname') def test_realpath_broken_symlinks(self): - ABSTFN = ntpath.abspath(support.TESTFN) + ABSTFN = ntpath.abspath(os_helper.TESTFN) os.mkdir(ABSTFN) self.addCleanup(support.rmtree, ABSTFN) @@ -335,18 +337,18 @@ def test_realpath_broken_symlinks(self): self.assertPathEqual(ntpath.realpath(b"broken5"), os.fsencode(ABSTFN + r"\missing")) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname') def test_realpath_symlink_loops(self): # Symlink loops are non-deterministic as to which path is returned, but # it will always be the fully resolved path of one member of the cycle - ABSTFN = ntpath.abspath(support.TESTFN) - self.addCleanup(support.unlink, ABSTFN) - self.addCleanup(support.unlink, ABSTFN + "1") - self.addCleanup(support.unlink, ABSTFN + "2") - self.addCleanup(support.unlink, ABSTFN + "y") - self.addCleanup(support.unlink, ABSTFN + "c") - self.addCleanup(support.unlink, ABSTFN + "a") + ABSTFN = ntpath.abspath(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, ABSTFN) + self.addCleanup(os_helper.unlink, ABSTFN + "1") + self.addCleanup(os_helper.unlink, ABSTFN + "2") + self.addCleanup(os_helper.unlink, ABSTFN + "y") + self.addCleanup(os_helper.unlink, ABSTFN + "c") + self.addCleanup(os_helper.unlink, ABSTFN + "a") os.symlink(ABSTFN, ABSTFN) self.assertPathEqual(ntpath.realpath(ABSTFN), ABSTFN) @@ -381,14 +383,14 @@ def test_realpath_symlink_loops(self): # Test using relative path as well. self.assertPathEqual(ntpath.realpath(ntpath.basename(ABSTFN)), ABSTFN) - @support.skip_unless_symlink + @os_helper.skip_unless_symlink @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname') def test_realpath_symlink_prefix(self): - ABSTFN = ntpath.abspath(support.TESTFN) - self.addCleanup(support.unlink, ABSTFN + "3") - self.addCleanup(support.unlink, "\\\\?\\" + ABSTFN + "3.") - self.addCleanup(support.unlink, ABSTFN + "3link") - self.addCleanup(support.unlink, ABSTFN + "3.link") + ABSTFN = ntpath.abspath(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, ABSTFN + "3") + self.addCleanup(os_helper.unlink, "\\\\?\\" + ABSTFN + "3.") + self.addCleanup(os_helper.unlink, ABSTFN + "3link") + self.addCleanup(os_helper.unlink, ABSTFN + "3.link") with open(ABSTFN + "3", "wb") as f: f.write(b'0') @@ -422,9 +424,9 @@ def test_realpath_nul(self): @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname') @unittest.skipUnless(HAVE_GETSHORTPATHNAME, 'need _getshortpathname') def test_realpath_cwd(self): - ABSTFN = ntpath.abspath(support.TESTFN) + ABSTFN = ntpath.abspath(os_helper.TESTFN) - support.unlink(ABSTFN) + os_helper.unlink(ABSTFN) support.rmtree(ABSTFN) os.mkdir(ABSTFN) self.addCleanup(support.rmtree, ABSTFN) @@ -449,7 +451,7 @@ def test_realpath_cwd(self): self.assertPathEqual(test_file_long, ntpath.realpath("file.txt")) def test_expandvars(self): - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.clear() env["foo"] = "bar" env["{foo"] = "baz1" @@ -474,13 +476,13 @@ def test_expandvars(self): tester('ntpath.expandvars("\'%foo%\'%bar")', "\'%foo%\'%bar") tester('ntpath.expandvars("bar\'%foo%")', "bar\'%foo%") - @unittest.skipUnless(support.FS_NONASCII, 'need support.FS_NONASCII') + @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') def test_expandvars_nonascii(self): def check(value, expected): tester('ntpath.expandvars(%r)' % value, expected) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.clear() - nonascii = support.FS_NONASCII + nonascii = os_helper.FS_NONASCII env['spam'] = nonascii env[nonascii] = 'ham' + nonascii check('$spam bar', '%s bar' % nonascii) @@ -497,7 +499,7 @@ def check(value, expected): def test_expanduser(self): tester('ntpath.expanduser("test")', 'test') - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.clear() tester('ntpath.expanduser("~test")', '~test') @@ -533,7 +535,7 @@ def test_expanduser(self): @unittest.skipUnless(nt, "abspath requires 'nt' module") def test_abspath(self): tester('ntpath.abspath("C:\\")', "C:\\") - with support.temp_cwd(support.TESTFN) as cwd_dir: # bpo-31047 + with os_helper.temp_cwd(os_helper.TESTFN) as cwd_dir: # bpo-31047 tester('ntpath.abspath("")', cwd_dir) tester('ntpath.abspath(" ")', cwd_dir + "\\ ") tester('ntpath.abspath("?")', cwd_dir + "\\?") @@ -545,7 +547,7 @@ def test_relpath(self): tester('ntpath.relpath(ntpath.abspath("a"))', 'a') tester('ntpath.relpath("a/b")', 'a\\b') tester('ntpath.relpath("../a/b")', '..\\a\\b') - with support.temp_cwd(support.TESTFN) as cwd_dir: + with os_helper.temp_cwd(os_helper.TESTFN) as cwd_dir: currentdir = ntpath.basename(cwd_dir) tester('ntpath.relpath("a", "../b")', '..\\'+currentdir+'\\a') tester('ntpath.relpath("a/b", "../c")', '..\\'+currentdir+'\\a\\b') @@ -661,7 +663,7 @@ def test_ismount(self): self.assertTrue(ntpath.ismount(b"\\\\.\\c:\\")) self.assertTrue(ntpath.ismount(b"\\\\.\\C:\\")) - with support.temp_dir() as d: + with os_helper.temp_dir() as d: self.assertFalse(ntpath.ismount(d)) if sys.platform == "win32": @@ -725,9 +727,9 @@ class PathLikeTests(NtpathTestCase): path = ntpath def setUp(self): - self.file_name = support.TESTFN - self.file_path = FakePath(support.TESTFN) - self.addCleanup(support.unlink, self.file_name) + self.file_name = os_helper.TESTFN + self.file_path = FakePath(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, self.file_name) with open(self.file_name, 'xb', 0) as file: file.write(b"test_ntpath.PathLikeTests") diff --git a/Lib/test/test_operator.py b/Lib/test/test_operator.py index 29f5e4275c55e..1ecae85f62f2c 100644 --- a/Lib/test/test_operator.py +++ b/Lib/test/test_operator.py @@ -3,9 +3,13 @@ import sys from test import support +from test.support import import_helper -py_operator = support.import_fresh_module('operator', blocked=['_operator']) -c_operator = support.import_fresh_module('operator', fresh=['_operator']) + +py_operator = import_helper.import_fresh_module('operator', + blocked=['_operator']) +c_operator = import_helper.import_fresh_module('operator', + fresh=['_operator']) class Seq1: def __init__(self, lst): diff --git a/Lib/test/test_optparse.py b/Lib/test/test_optparse.py index 437fdd2be8d85..ed65b7798e3d2 100644 --- a/Lib/test/test_optparse.py +++ b/Lib/test/test_optparse.py @@ -14,6 +14,7 @@ from io import StringIO from test import support +from test.support import os_helper import optparse @@ -1021,10 +1022,10 @@ def setUp(self): self.parser.add_option("-f", "--file", type="file", dest="file") def tearDown(self): - if os.path.isdir(support.TESTFN): - os.rmdir(support.TESTFN) - elif os.path.isfile(support.TESTFN): - os.unlink(support.TESTFN) + if os.path.isdir(os_helper.TESTFN): + os.rmdir(os_helper.TESTFN) + elif os.path.isfile(os_helper.TESTFN): + os.unlink(os_helper.TESTFN) class MyOption (Option): def check_file(option, opt, value): @@ -1039,21 +1040,21 @@ def check_file(option, opt, value): TYPE_CHECKER["file"] = check_file def test_filetype_ok(self): - support.create_empty_file(support.TESTFN) - self.assertParseOK(["--file", support.TESTFN, "-afoo"], - {'file': support.TESTFN, 'a': 'foo'}, + os_helper.create_empty_file(os_helper.TESTFN) + self.assertParseOK(["--file", os_helper.TESTFN, "-afoo"], + {'file': os_helper.TESTFN, 'a': 'foo'}, []) def test_filetype_noexist(self): - self.assertParseFail(["--file", support.TESTFN, "-afoo"], + self.assertParseFail(["--file", os_helper.TESTFN, "-afoo"], "%s: file does not exist" % - support.TESTFN) + os_helper.TESTFN) def test_filetype_notfile(self): - os.mkdir(support.TESTFN) - self.assertParseFail(["--file", support.TESTFN, "-afoo"], + os.mkdir(os_helper.TESTFN) + self.assertParseFail(["--file", os_helper.TESTFN, "-afoo"], "%s: not a regular file" % - support.TESTFN) + os_helper.TESTFN) class TestExtendAddActions(BaseTest): @@ -1497,7 +1498,7 @@ def make_parser(self, columns): # we must restore its original value -- otherwise, this test # screws things up for other tests when it's part of the Python # test suite. - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env['COLUMNS'] = str(columns) return InterceptingOptionParser(option_list=options) @@ -1522,7 +1523,7 @@ def test_help_long_opts_first(self): self.assertHelpEquals(_expected_help_long_opts_first) def test_help_title_formatter(self): - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["COLUMNS"] = "80" self.parser.formatter = TitledHelpFormatter() self.assertHelpEquals(_expected_help_title_formatter) diff --git a/Lib/test/test_peg_generator/test_c_parser.py b/Lib/test/test_peg_generator/test_c_parser.py index f9935258c861e..2c13635d37dea 100644 --- a/Lib/test/test_peg_generator/test_c_parser.py +++ b/Lib/test/test_peg_generator/test_c_parser.py @@ -5,6 +5,7 @@ from test import test_tools from test import support +from test.support import os_helper from test.support.script_helper import assert_python_ok test_tools.skip_if_missing("peg_generator") @@ -68,7 +69,7 @@ def setUp(self): self.skipTest("The %r command is not found" % cmd) super(TestCParser, self).setUp() self.tmp_path = self.mkdtemp() - change_cwd = support.change_cwd(self.tmp_path) + change_cwd = os_helper.change_cwd(self.tmp_path) change_cwd.__enter__() self.addCleanup(change_cwd.__exit__, None, None, None) diff --git a/Lib/test/test_queue.py b/Lib/test/test_queue.py index 7b23699a00f1d..508b739019593 100644 --- a/Lib/test/test_queue.py +++ b/Lib/test/test_queue.py @@ -6,11 +6,12 @@ import time import unittest import weakref -from test import support +from test.support import import_helper from test.support import threading_helper -py_queue = support.import_fresh_module('queue', blocked=['_queue']) -c_queue = support.import_fresh_module('queue', fresh=['_queue']) + +py_queue = import_helper.import_fresh_module('queue', blocked=['_queue']) +c_queue = import_helper.import_fresh_module('queue', fresh=['_queue']) need_c_queue = unittest.skipUnless(c_queue, "No _queue module found") QUEUE_SIZE = 5 diff --git a/Lib/test/test_spwd.py b/Lib/test/test_spwd.py index 07793c84c8e91..a143acc659ef6 100644 --- a/Lib/test/test_spwd.py +++ b/Lib/test/test_spwd.py @@ -1,8 +1,9 @@ import os import unittest -from test import support +from test.support import import_helper -spwd = support.import_module('spwd') + +spwd = import_helper.import_module('spwd') @unittest.skipUnless(hasattr(os, 'geteuid') and os.geteuid() == 0, diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py index d9e4ffde658db..83d09e17f93c5 100644 --- a/Lib/test/test_stat.py +++ b/Lib/test/test_stat.py @@ -3,7 +3,9 @@ import socket import sys from test.support import socket_helper -from test.support import TESTFN, import_fresh_module +from test.support.import_helper import import_fresh_module +from test.support.os_helper import TESTFN + c_stat = import_fresh_module('stat', fresh=['_stat']) py_stat = import_fresh_module('stat', blocked=['_stat']) diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py index 6de7aa87bb2f9..681f2c72f9c37 100644 --- a/Lib/test/test_tokenize.py +++ b/Lib/test/test_tokenize.py @@ -1,4 +1,5 @@ from test import support +from test.support import os_helper from tokenize import (tokenize, _tokenize, untokenize, NUMBER, NAME, OP, STRING, ENDMARKER, ENCODING, tok_name, detect_encoding, open as tokenize_open, Untokenizer, generate_tokens, @@ -1265,8 +1266,8 @@ def test_false_encoding(self): self.assertEqual(consumed_lines, [b'print("#coding=fake")']) def test_open(self): - filename = support.TESTFN + '.py' - self.addCleanup(support.unlink, filename) + filename = os_helper.TESTFN + '.py' + self.addCleanup(os_helper.unlink, filename) # test coding cookie for encoding in ('iso-8859-15', 'utf-8'): diff --git a/Lib/test/test_webbrowser.py b/Lib/test/test_webbrowser.py index 519a9432abe01..6ceb49069f656 100644 --- a/Lib/test/test_webbrowser.py +++ b/Lib/test/test_webbrowser.py @@ -5,6 +5,7 @@ import subprocess from unittest import mock from test import support +from test.support import import_helper URL = 'http://www.example.com' @@ -270,7 +271,7 @@ def test_register_preferred(self): class ImportTest(unittest.TestCase): def test_register(self): - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') self.assertIsNone(webbrowser._tryorder) self.assertFalse(webbrowser._browsers) @@ -284,7 +285,7 @@ class ExampleBrowser: self.assertEqual(webbrowser._browsers['example1'], [ExampleBrowser, None]) def test_get(self): - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') self.assertIsNone(webbrowser._tryorder) self.assertFalse(webbrowser._browsers) @@ -293,24 +294,24 @@ def test_get(self): self.assertIsNotNone(webbrowser._tryorder) def test_synthesize(self): - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') name = os.path.basename(sys.executable).lower() webbrowser.register(name, None, webbrowser.GenericBrowser(name)) webbrowser.get(sys.executable) def test_environment(self): - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') try: browser = webbrowser.get().name except (webbrowser.Error, AttributeError) as err: self.skipTest(str(err)) with support.EnvironmentVarGuard() as env: env["BROWSER"] = browser - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') webbrowser.get() def test_environment_preferred(self): - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') try: webbrowser.get() least_preferred_browser = webbrowser.get(webbrowser._tryorder[-1]).name @@ -319,12 +320,12 @@ def test_environment_preferred(self): with support.EnvironmentVarGuard() as env: env["BROWSER"] = least_preferred_browser - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') self.assertEqual(webbrowser.get().name, least_preferred_browser) with support.EnvironmentVarGuard() as env: env["BROWSER"] = sys.executable - webbrowser = support.import_fresh_module('webbrowser') + webbrowser = import_helper.import_fresh_module('webbrowser') self.assertEqual(webbrowser.get().name, sys.executable) From webhook-mailer at python.org Mon Aug 3 16:51:46 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 03 Aug 2020 20:51:46 -0000 Subject: [Python-checkins] bpo-38912: regrtest logs unraisable exception into sys.__stderr__ (GH-21718) Message-ID: https://github.com/python/cpython/commit/701b63894fdb75b12865b9be6261ce4913da76f5 commit: 701b63894fdb75b12865b9be6261ce4913da76f5 branch: master author: Victor Stinner committer: GitHub date: 2020-08-03T22:51:23+02:00 summary: bpo-38912: regrtest logs unraisable exception into sys.__stderr__ (GH-21718) regrtest_unraisable_hook() temporarily replaces sys.stderr with sys.__stderr__ to help to display errors when a test captures stderr. files: M Lib/test/libregrtest/utils.py M Lib/test/test_regrtest.py diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index 0368694b2adcb..71f538f0c4533 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -72,7 +72,12 @@ def regrtest_unraisable_hook(unraisable): global orig_unraisablehook support.environment_altered = True print_warning("Unraisable exception") - orig_unraisablehook(unraisable) + old_stderr = sys.stderr + try: + sys.stderr = sys.__stderr__ + orig_unraisablehook(unraisable) + finally: + sys.stderr = old_stderr def setup_unraisable_hook(): diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index 39af0d96d1e54..38321e04b54a9 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -1235,10 +1235,12 @@ def test_sleep(self): re.compile('%s timed out' % testname, re.MULTILINE)) def test_unraisable_exc(self): - # --fail-env-changed must catch unraisable exception + # --fail-env-changed must catch unraisable exception. + # The exceptioin must be displayed even if sys.stderr is redirected. code = textwrap.dedent(r""" import unittest import weakref + from test.support import captured_stderr class MyObject: pass @@ -1250,9 +1252,11 @@ class Tests(unittest.TestCase): def test_unraisable_exc(self): obj = MyObject() ref = weakref.ref(obj, weakref_callback) - # call weakref_callback() which logs - # an unraisable exception - obj = None + with captured_stderr() as stderr: + # call weakref_callback() which logs + # an unraisable exception + obj = None + self.assertEqual(stderr.getvalue(), '') """) testname = self.create_test(code=code) @@ -1261,6 +1265,7 @@ def test_unraisable_exc(self): env_changed=[testname], fail_env_changed=True) self.assertIn("Warning -- Unraisable exception", output) + self.assertIn("Exception: weakref callback bug", output) def test_cleanup(self): dirname = os.path.join(self.tmptestdir, "test_python_123") From webhook-mailer at python.org Mon Aug 3 20:38:24 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 04 Aug 2020 00:38:24 -0000 Subject: [Python-checkins] bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721) Message-ID: https://github.com/python/cpython/commit/bde48fd8110cc5f128d5db44810d17811e328a24 commit: bde48fd8110cc5f128d5db44810d17811e328a24 branch: master author: Victor Stinner committer: GitHub date: 2020-08-04T02:38:16+02:00 summary: bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721) incr cannot be larger than INT_MAX: downcast to int explicitly. files: M Parser/myreadline.c diff --git a/Parser/myreadline.c b/Parser/myreadline.c index a49c9d892dda9..143b41f1eab95 100644 --- a/Parser/myreadline.c +++ b/Parser/myreadline.c @@ -317,7 +317,7 @@ PyOS_StdioReadline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt) return NULL; } p = pr; - int err = my_fgets(tstate, p + n, incr, sys_stdin); + int err = my_fgets(tstate, p + n, (int)incr, sys_stdin); if (err == 1) { // Interrupt PyMem_RawFree(p); From webhook-mailer at python.org Mon Aug 3 20:40:14 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 04 Aug 2020 00:40:14 -0000 Subject: [Python-checkins] bpo-41467: Fix asyncio recv_into() on Windows (GH-21720) Message-ID: https://github.com/python/cpython/commit/602a971a2af3a685d625c912c400cadd452718b1 commit: 602a971a2af3a685d625c912c400cadd452718b1 branch: master author: Victor Stinner committer: GitHub date: 2020-08-04T02:40:10+02:00 summary: bpo-41467: Fix asyncio recv_into() on Windows (GH-21720) On Windows, fix asyncio recv_into() return value when the socket/pipe is closed (BrokenPipeError): return 0 rather than an empty byte string (b''). files: A Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst M Lib/asyncio/windows_events.py diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py index c07fe3241c569..a6759b78bd801 100644 --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -469,7 +469,7 @@ def recv_into(self, conn, buf, flags=0): else: ov.ReadFileInto(conn.fileno(), buf) except BrokenPipeError: - return self._result(b'') + return self._result(0) def finish_recv(trans, key, ov): try: diff --git a/Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst b/Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst new file mode 100644 index 0000000000000..f12693e117631 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst @@ -0,0 +1,3 @@ +On Windows, fix asyncio ``recv_into()`` return value when the socket/pipe is +closed (:exc:`BrokenPipeError`): return ``0`` rather than an empty byte +string (``b''``). From webhook-mailer at python.org Mon Aug 3 20:58:10 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 04 Aug 2020 00:58:10 -0000 Subject: [Python-checkins] bpo-41467: Fix asyncio recv_into() on Windows (GH-21720) Message-ID: https://github.com/python/cpython/commit/b934d832d1e16bf235c536dcde3006faf29757fc commit: b934d832d1e16bf235c536dcde3006faf29757fc branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-03T17:58:06-07:00 summary: bpo-41467: Fix asyncio recv_into() on Windows (GH-21720) On Windows, fix asyncio recv_into() return value when the socket/pipe is closed (BrokenPipeError): return 0 rather than an empty byte string (b''). (cherry picked from commit 602a971a2af3a685d625c912c400cadd452718b1) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst M Lib/asyncio/windows_events.py diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py index ac51109ff1a83..12e87abfbf2ea 100644 --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -469,7 +469,7 @@ def recv_into(self, conn, buf, flags=0): else: ov.ReadFileInto(conn.fileno(), buf) except BrokenPipeError: - return self._result(b'') + return self._result(0) def finish_recv(trans, key, ov): try: diff --git a/Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst b/Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst new file mode 100644 index 0000000000000..f12693e117631 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-04-00-20-30.bpo-41467.Z8DgTL.rst @@ -0,0 +1,3 @@ +On Windows, fix asyncio ``recv_into()`` return value when the socket/pipe is +closed (:exc:`BrokenPipeError`): return ``0`` rather than an empty byte +string (``b''``). From webhook-mailer at python.org Mon Aug 3 20:59:06 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 04 Aug 2020 00:59:06 -0000 Subject: [Python-checkins] bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721) Message-ID: https://github.com/python/cpython/commit/46e448abbf35c051e5306a4695670d7ec7abc4e9 commit: 46e448abbf35c051e5306a4695670d7ec7abc4e9 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-03T17:59:02-07:00 summary: bpo-38156: Fix compiler warning in PyOS_StdioReadline() (GH-21721) incr cannot be larger than INT_MAX: downcast to int explicitly. (cherry picked from commit bde48fd8110cc5f128d5db44810d17811e328a24) Co-authored-by: Victor Stinner files: M Parser/myreadline.c diff --git a/Parser/myreadline.c b/Parser/myreadline.c index 29b254f771f16..e8e57738f3377 100644 --- a/Parser/myreadline.c +++ b/Parser/myreadline.c @@ -318,7 +318,7 @@ PyOS_StdioReadline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt) return NULL; } p = pr; - int err = my_fgets(tstate, p + n, incr, sys_stdin); + int err = my_fgets(tstate, p + n, (int)incr, sys_stdin); if (err == 1) { // Interrupt PyMem_RawFree(p); From webhook-mailer at python.org Mon Aug 3 22:08:32 2020 From: webhook-mailer at python.org (Inada Naoki) Date: Tue, 04 Aug 2020 02:08:32 -0000 Subject: [Python-checkins] bpo-41431: Optimize dict_merge for copy (GH-21674) Message-ID: https://github.com/python/cpython/commit/db6d9a50cee92c0ded7c5cb87331c5f0b1008698 commit: db6d9a50cee92c0ded7c5cb87331c5f0b1008698 branch: master author: Inada Naoki committer: GitHub date: 2020-08-04T11:08:06+09:00 summary: bpo-41431: Optimize dict_merge for copy (GH-21674) files: A Misc/NEWS.d/next/Core and Builtins/2020-08-02-15-53-12.bpo-41431.TblUBT.rst M Lib/test/test_ordered_dict.py M Objects/dictobject.c diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py index 1f27c0e0b57eb..31759f20d2834 100644 --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -740,20 +740,21 @@ def test_sizeof_exact(self): size = support.calcobjsize check = self.check_sizeof - basicsize = size('nQ2P' + '3PnPn2P') + calcsize('2nP2n') + basicsize = size('nQ2P' + '3PnPn2P') + keysize = calcsize('2nP2n') entrysize = calcsize('n2P') p = calcsize('P') nodesize = calcsize('Pn2P') od = OrderedDict() - check(od, basicsize + 8 + 5*entrysize) # 8byte indices + 8*2//3 * entry table + check(od, basicsize) # 8byte indices + 8*2//3 * entry table od.x = 1 - check(od, basicsize + 8 + 5*entrysize) + check(od, basicsize) od.update([(i, i) for i in range(3)]) - check(od, basicsize + 8*p + 8 + 5*entrysize + 3*nodesize) + check(od, basicsize + keysize + 8*p + 8 + 5*entrysize + 3*nodesize) od.update([(i, i) for i in range(3, 10)]) - check(od, basicsize + 16*p + 16 + 10*entrysize + 10*nodesize) + check(od, basicsize + keysize + 16*p + 16 + 10*entrysize + 10*nodesize) check(od.keys(), size('P')) check(od.items(), size('P')) diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-02-15-53-12.bpo-41431.TblUBT.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-02-15-53-12.bpo-41431.TblUBT.rst new file mode 100644 index 0000000000000..fa9d047edc394 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-02-15-53-12.bpo-41431.TblUBT.rst @@ -0,0 +1,2 @@ +Optimize ``dict_merge()`` for copying dict (e.g. ``dict(d)`` and +``{}.update(d)``). diff --git a/Objects/dictobject.c b/Objects/dictobject.c index ba22489539ae7..1b7ae06d82271 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -674,10 +674,11 @@ new_dict_with_shared_keys(PyDictKeysObject *keys) } -static PyObject * -clone_combined_dict(PyDictObject *orig) +static PyDictKeysObject * +clone_combined_dict_keys(PyDictObject *orig) { - assert(PyDict_CheckExact(orig)); + assert(PyDict_Check(orig)); + assert(Py_TYPE(orig)->tp_iter == (getiterfunc)dict_iter); assert(orig->ma_values == NULL); assert(orig->ma_keys->dk_refcnt == 1); @@ -704,19 +705,6 @@ clone_combined_dict(PyDictObject *orig) } } - PyDictObject *new = (PyDictObject *)new_dict(keys, NULL); - if (new == NULL) { - /* In case of an error, `new_dict()` takes care of - cleaning up `keys`. */ - return NULL; - } - new->ma_used = orig->ma_used; - ASSERT_CONSISTENT(new); - if (_PyObject_GC_IS_TRACKED(orig)) { - /* Maintain tracking. */ - _PyObject_GC_TRACK(new); - } - /* Since we copied the keys table we now have an extra reference in the system. Manually call increment _Py_RefTotal to signal that we have it now; calling dictkeys_incref would be an error as @@ -724,8 +712,7 @@ clone_combined_dict(PyDictObject *orig) #ifdef Py_REF_DEBUG _Py_RefTotal++; #endif - - return (PyObject *)new; + return keys; } PyObject * @@ -2527,12 +2514,45 @@ dict_merge(PyObject *a, PyObject *b, int override) if (other == mp || other->ma_used == 0) /* a.update(a) or a.update({}); nothing to do */ return 0; - if (mp->ma_used == 0) + if (mp->ma_used == 0) { /* Since the target dict is empty, PyDict_GetItem() * always returns NULL. Setting override to 1 * skips the unnecessary test. */ override = 1; + PyDictKeysObject *okeys = other->ma_keys; + + // If other is clean, combined, and just allocated, just clone it. + if (other->ma_values == NULL && + other->ma_used == okeys->dk_nentries && + (okeys->dk_size == PyDict_MINSIZE || + USABLE_FRACTION(okeys->dk_size/2) < other->ma_used)) { + PyDictKeysObject *keys = clone_combined_dict_keys(other); + if (keys == NULL) { + return -1; + } + + dictkeys_decref(mp->ma_keys); + mp->ma_keys = keys; + if (mp->ma_values != NULL) { + if (mp->ma_values != empty_values) { + free_values(mp->ma_values); + } + mp->ma_values = NULL; + } + + mp->ma_used = other->ma_used; + mp->ma_version_tag = DICT_NEXT_VERSION(); + ASSERT_CONSISTENT(mp); + + if (_PyObject_GC_IS_TRACKED(other) && !_PyObject_GC_IS_TRACKED(mp)) { + /* Maintain tracking. */ + _PyObject_GC_TRACK(mp); + } + + return 0; + } + } /* Do one big resize at the start, rather than * incrementally resizing as we insert new items. Expect * that there will be no (or few) overlapping keys. @@ -2718,12 +2738,13 @@ PyDict_Copy(PyObject *o) return (PyObject *)split_copy; } - if (PyDict_CheckExact(mp) && mp->ma_values == NULL && + if (Py_TYPE(mp)->tp_iter == (getiterfunc)dict_iter && + mp->ma_values == NULL && (mp->ma_used >= (mp->ma_keys->dk_nentries * 2) / 3)) { /* Use fast-copy if: - (1) 'mp' is an instance of a subclassed dict; and + (1) type(mp) doesn't override tp_iter; and (2) 'mp' is not a split-dict; and @@ -2735,13 +2756,31 @@ PyDict_Copy(PyObject *o) operations and copied after that. In cases like this, we defer to PyDict_Merge, which produces a compacted copy. */ - return clone_combined_dict(mp); + PyDictKeysObject *keys = clone_combined_dict_keys(mp); + if (keys == NULL) { + return NULL; + } + PyDictObject *new = (PyDictObject *)new_dict(keys, NULL); + if (new == NULL) { + /* In case of an error, `new_dict()` takes care of + cleaning up `keys`. */ + return NULL; + } + + new->ma_used = mp->ma_used; + ASSERT_CONSISTENT(new); + if (_PyObject_GC_IS_TRACKED(mp)) { + /* Maintain tracking. */ + _PyObject_GC_TRACK(new); + } + + return (PyObject *)new; } copy = PyDict_New(); if (copy == NULL) return NULL; - if (PyDict_Merge(copy, o, 1) == 0) + if (dict_merge(copy, o, 1) == 0) return copy; Py_DECREF(copy); return NULL; @@ -3359,16 +3398,15 @@ dict_new(PyTypeObject *type, PyObject *args, PyObject *kwds) d = (PyDictObject *)self; /* The object has been implicitly tracked by tp_alloc */ - if (type == &PyDict_Type) + if (type == &PyDict_Type) { _PyObject_GC_UNTRACK(d); + } d->ma_used = 0; d->ma_version_tag = DICT_NEXT_VERSION(); - d->ma_keys = new_keys_object(PyDict_MINSIZE); - if (d->ma_keys == NULL) { - Py_DECREF(self); - return NULL; - } + dictkeys_incref(Py_EMPTY_KEYS); + d->ma_keys = Py_EMPTY_KEYS; + d->ma_values = empty_values; ASSERT_CONSISTENT(d); return self; } From webhook-mailer at python.org Mon Aug 3 22:16:29 2020 From: webhook-mailer at python.org (Steve Dower) Date: Tue, 04 Aug 2020 02:16:29 -0000 Subject: [Python-checkins] [3.5] bpo-29778: Ensure python3.dll is loaded from correct locations when Python is embedded (GH-21297) (#21377) Message-ID: https://github.com/python/cpython/commit/f205f1000a2d7f8b044caf281041b3705f293480 commit: f205f1000a2d7f8b044caf281041b3705f293480 branch: 3.5 author: Steve Dower committer: GitHub date: 2020-08-03T19:16:20-07:00 summary: [3.5] bpo-29778: Ensure python3.dll is loaded from correct locations when Python is embedded (GH-21297) (#21377) bpo-29778: Ensure python3.dll is loaded from correct locations when Python is embedded. files: A Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst M PC/getpathp.c M PCbuild/pyproject.props M PCbuild/python.props M Python/dynload_win.c diff --git a/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst b/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst new file mode 100644 index 0000000000000..998ffb1ee6667 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst @@ -0,0 +1,2 @@ +Ensure :file:`python3.dll` is loaded from correct locations when Python is +embedded (CVE-2020-15523). diff --git a/PC/getpathp.c b/PC/getpathp.c index 47a571e272b00..282d3bc4f5f7a 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -93,7 +93,6 @@ static wchar_t prefix[MAXPATHLEN+1]; static wchar_t progpath[MAXPATHLEN+1]; -static wchar_t dllpath[MAXPATHLEN+1]; static wchar_t *module_search_path = NULL; @@ -122,6 +121,46 @@ reduce(wchar_t *dir) dir[i] = '\0'; } +static int +change_ext(wchar_t *dest, const wchar_t *src, const wchar_t *ext) +{ + if (src && src != dest) { + size_t src_len = wcsnlen_s(src, MAXPATHLEN+1); + size_t i = src_len; + if (i >= MAXPATHLEN+1) { + Py_FatalError("buffer overflow in getpathp.c's reduce()"); + } + + while (i > 0 && src[i] != '.' && !is_sep(src[i])) + --i; + + if (i == 0) { + dest[0] = '\0'; + return -1; + } + + if (is_sep(src[i])) { + i = src_len; + } + + if (wcsncpy_s(dest, MAXPATHLEN+1, src, i)) { + dest[0] = '\0'; + return -1; + } + } else { + wchar_t *s = wcsrchr(dest, L'.'); + if (s) { + s[0] = '\0'; + } + } + + if (wcscat_s(dest, MAXPATHLEN+1, ext)) { + dest[0] = '\0'; + return -1; + } + + return 0; +} static int exists(wchar_t *filename) @@ -214,6 +253,20 @@ search_for_prefix(wchar_t *argv0_path, wchar_t *landmark) return 0; } + +static int +get_dllpath(wchar_t *dllpath) +{ +#ifdef Py_ENABLE_SHARED + extern HANDLE PyWin_DLLhModule; + if (PyWin_DLLhModule && GetModuleFileNameW(PyWin_DLLhModule, dllpath, MAXPATHLEN)) { + return 0; + } +#endif + return -1; +} + + #ifdef MS_WINDOWS #ifdef Py_ENABLE_SHARED @@ -378,19 +431,8 @@ get_progpath(void) wchar_t *path = _wgetenv(L"PATH"); wchar_t *prog = Py_GetProgramName(); -#ifdef MS_WINDOWS -#ifdef Py_ENABLE_SHARED - extern HANDLE PyWin_DLLhModule; - /* static init of progpath ensures final char remains \0 */ - if (PyWin_DLLhModule) - if (!GetModuleFileNameW(PyWin_DLLhModule, dllpath, MAXPATHLEN)) - dllpath[0] = 0; -#else - dllpath[0] = 0; -#endif if (GetModuleFileNameW(NULL, progpath, MAXPATHLEN)) return; -#endif if (prog == NULL || *prog == '\0') prog = L"python"; @@ -578,14 +620,10 @@ calculate_path(void) #ifdef MS_WINDOWS /* Calculate zip archive path from DLL or exe path */ - if (wcscpy_s(zip_path, MAXPATHLEN+1, dllpath[0] ? dllpath : progpath)) - /* exceeded buffer length - ignore zip_path */ - zip_path[0] = '\0'; - else { - wchar_t *dot = wcsrchr(zip_path, '.'); - if (!dot || wcscpy_s(dot, MAXPATHLEN+1 - (dot - zip_path), L".zip")) - /* exceeded buffer length - ignore zip_path */ - zip_path[0] = L'\0'; + if (!get_dllpath(zip_path)) { + change_ext(zip_path, zip_path, L".zip"); + } else { + change_ext(zip_path, progpath, L".zip"); } skiphome = pythonhome==NULL ? 0 : 1; @@ -768,8 +806,6 @@ calculate_path(void) } -/* External interface */ - void Py_SetPath(const wchar_t *path) { @@ -830,25 +866,39 @@ int _Py_CheckPython3() { wchar_t py3path[MAXPATHLEN+1]; - wchar_t *s; - if (python3_checked) + if (python3_checked) { return hPython3 != NULL; + } python3_checked = 1; /* If there is a python3.dll next to the python3y.dll, - assume this is a build tree; use that DLL */ - wcscpy(py3path, dllpath); - s = wcsrchr(py3path, L'\\'); - if (!s) - s = py3path; - wcscpy(s, L"\\python3.dll"); - hPython3 = LoadLibraryExW(py3path, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); - if (hPython3 != NULL) - return 1; + use that DLL */ + if (!get_dllpath(py3path)) { + reduce(py3path); + join(py3path, PY3_DLLNAME); + hPython3 = LoadLibraryExW(py3path, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); + if (hPython3 != NULL) { + return 1; + } + } - /* Check sys.prefix\DLLs\python3.dll */ + /* If we can locate python3.dll in our application dir, + use that DLL */ wcscpy(py3path, Py_GetPrefix()); - wcscat(py3path, L"\\DLLs\\python3.dll"); - hPython3 = LoadLibraryExW(py3path, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); + if (py3path[0]) { + join(py3path, PY3_DLLNAME); + hPython3 = LoadLibraryExW(py3path, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); + if (hPython3 != NULL) { + return 1; + } + } + + /* For back-compat, also search {sys.prefix}\DLLs, though + that has not been a normal install layout for a while */ + wcscpy(py3path, Py_GetPrefix()); + if (py3path[0]) { + join(py3path, L"DLLs\\" PY3_DLLNAME); + hPython3 = LoadLibraryExW(py3path, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); + } return hPython3 != NULL; } diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props index d1ac99847b980..03197fe63772a 100644 --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -24,12 +24,12 @@ <_PlatformPreprocessorDefinition>_WIN32; <_PlatformPreprocessorDefinition Condition="$(Platform) == 'x64'">_WIN64;_M_X64; <_PydPreprocessorDefinition Condition="$(TargetExt) == '.pyd'">Py_BUILD_CORE_MODULE; + <_Py3NamePreprocessorDefinition>PY3_DLLNAME=L"$(Py3DllName)"; $(PySourcePath)Include;$(PySourcePath)PC;$(IntDir);%(AdditionalIncludeDirectories) - WIN32;$(_PlatformPreprocessorDefinition)$(_DebugPreprocessorDefinition)$(_PydPreprocessorDefinition)%(PreprocessorDefinitions) - + WIN32;$(_Py3NamePreprocessorDefinition)$(_PlatformPreprocessorDefinition)$(_DebugPreprocessorDefinition)$(_PydPreprocessorDefinition)%(PreprocessorDefinitions) MaxSpeed true true diff --git a/PCbuild/python.props b/PCbuild/python.props index 0a6fd2a651382..f59bcea5d1d28 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -152,6 +152,8 @@ python$(MajorVersionNumber)$(MinorVersionNumber)$(PyDebugExt) + + python3$(PyDebugExt) .cp$(MajorVersionNumber)$(MinorVersionNumber)-win32 diff --git a/Python/dynload_win.c b/Python/dynload_win.c index 3f533d1756cf2..1be876d83e811 100644 --- a/Python/dynload_win.c +++ b/Python/dynload_win.c @@ -194,9 +194,7 @@ dl_funcptr _PyImport_FindSharedFuncptrWindows(const char *prefix, char funcname[258], *import_python; const wchar_t *wpathname; -#ifndef _DEBUG _Py_CheckPython3(); -#endif wpathname = _PyUnicode_AsUnicode(pathname); if (wpathname == NULL) From webhook-mailer at python.org Mon Aug 3 22:33:36 2020 From: webhook-mailer at python.org (Tapas Kundu) Date: Tue, 04 Aug 2020 02:33:36 -0000 Subject: [Python-checkins] [3.5] bpo-41004: Resolve hash collisions for IPv4Interface and IPv6Interface (GH-21033) (#21233) Message-ID: https://github.com/python/cpython/commit/11d258ceafdf60ab3840f9a5700f2d0ad3e2e2d1 commit: 11d258ceafdf60ab3840f9a5700f2d0ad3e2e2d1 branch: 3.5 author: Tapas Kundu <39723251+tapakund at users.noreply.github.com> committer: GitHub date: 2020-08-03T19:33:30-07:00 summary: [3.5] bpo-41004: Resolve hash collisions for IPv4Interface and IPv6Interface (GH-21033) (#21233) CVE-2020-14422: The __hash__() methods of classes IPv4Interface and IPv6Interface had issue of generating constant hash values of 32 and 128 respectively causing hash collisions. The fix uses the hash() function to generate hash values for the objects instead of XOR operation. (cherry picked from commit b30ee26e366bf509b7538d79bfec6c6d38d53f28) Co-authored-by: Ravi Teja P Signed-off-by: Tapas Kundu files: A Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst M Lib/ipaddress.py M Lib/test/test_ipaddress.py diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index 35ca38e54fe21..b35c490d7b0d7 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1418,7 +1418,7 @@ def __lt__(self, other): return False def __hash__(self): - return self._ip ^ self._prefixlen ^ int(self.network.network_address) + return hash((self._ip, self._prefixlen, int(self.network.network_address))) __reduce__ = _IPAddressBase.__reduce__ @@ -2109,7 +2109,7 @@ def __lt__(self, other): return False def __hash__(self): - return self._ip ^ self._prefixlen ^ int(self.network.network_address) + return hash((self._ip, self._prefixlen, int(self.network.network_address))) __reduce__ = _IPAddressBase.__reduce__ diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py index e3dec87ed990f..92c0812f6e982 100644 --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -1966,6 +1966,17 @@ def testsixtofour(self): sixtofouraddr.sixtofour) self.assertFalse(bad_addr.sixtofour) + # issue41004 Hash collisions in IPv4Interface and IPv6Interface + def testV4HashIsNotConstant(self): + ipv4_address1 = ipaddress.IPv4Interface("1.2.3.4") + ipv4_address2 = ipaddress.IPv4Interface("2.3.4.5") + self.assertNotEqual(ipv4_address1.__hash__(), ipv4_address2.__hash__()) + + # issue41004 Hash collisions in IPv4Interface and IPv6Interface + def testV6HashIsNotConstant(self): + ipv6_address1 = ipaddress.IPv6Interface("2001:658:22a:cafe:200:0:0:1") + ipv6_address2 = ipaddress.IPv6Interface("2001:658:22a:cafe:200:0:0:2") + self.assertNotEqual(ipv6_address1.__hash__(), ipv6_address2.__hash__()) if __name__ == '__main__': unittest.main() diff --git a/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst b/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst new file mode 100644 index 0000000000000..f5a9db52fff52 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst @@ -0,0 +1 @@ +CVE-2020-14422: The __hash__() methods of ipaddress.IPv4Interface and ipaddress.IPv6Interface incorrectly generated constant hash values of 32 and 128 respectively. This resulted in always causing hash collisions. The fix uses hash() to generate hash values for the tuple of (address, mask length, network address). From webhook-mailer at python.org Mon Aug 3 23:51:44 2020 From: webhook-mailer at python.org (Hans Petter Jansson) Date: Tue, 04 Aug 2020 03:51:44 -0000 Subject: [Python-checkins] bpo-36982: Add support for extended color functions in ncurses 6.1 (GH-17536) Message-ID: https://github.com/python/cpython/commit/da4e09fff6b483fe858997da5599c25397107ca1 commit: da4e09fff6b483fe858997da5599c25397107ca1 branch: master author: Hans Petter Jansson committer: GitHub date: 2020-08-03T23:51:33-04:00 summary: bpo-36982: Add support for extended color functions in ncurses 6.1 (GH-17536) Co-authored-by: Jeffrey Kintscher files: A Misc/NEWS.d/next/Core and Builtins/2019-05-25-05-27-39.bpo-36982.0UHgfB.rst M Doc/library/curses.rst M Doc/whatsnew/3.10.rst M Lib/test/test_curses.py M Misc/ACKS M Modules/_cursesmodule.c M Modules/clinic/_cursesmodule.c.h diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index c86ca5d9bc554..0b687db1bd2c4 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -242,6 +242,15 @@ The module :mod:`curses` defines the following functions: Return ``True`` if the terminal can display colors; otherwise, return ``False``. +.. function:: has_extended_color_support() + + Return ``True`` if the module supports extended colors; otherwise, return + ``False``. Extended color support allows more than 256 color pairs for + terminals that support more than 16 colors (e.g. xterm-256color). + + Extended color support requires ncurses version 6.1 or later. + + .. versionadded:: 3.10 .. function:: has_ic() diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 1865fa227534a..ec0343f2ce71e 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -103,6 +103,16 @@ New Modules Improved Modules ================ +curses +------ + +The extended color functions added in ncurses 6.1 will be used transparently +by :func:`curses.color_content`, :func:`curses.init_color`, +:func:`curses.init_pair`, and :func:`curses.pair_content`. A new function, +:func:`curses.has_extended_color_support`, indicates whether extended color +support is provided by the underlying ncurses library. +(Contributed by Jeffrey Kintscher and Hans Petter Jansson in :issue:`36982`.) + glob ---- diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 2c6d14c3f79dd..cabc10da8365c 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -232,7 +232,8 @@ def test_module_funcs(self): curses.nocbreak, curses.noecho, curses.nonl, curses.noqiflush, curses.noraw, curses.reset_prog_mode, curses.termattrs, - curses.termname, curses.erasechar]: + curses.termname, curses.erasechar, + curses.has_extended_color_support]: with self.subTest(func=func.__qualname__): func() if hasattr(curses, 'filter'): @@ -293,6 +294,19 @@ def test_colors_funcs(self): if hasattr(curses, 'use_default_colors'): curses.use_default_colors() + self.assertRaises(ValueError, curses.color_content, -1) + self.assertRaises(ValueError, curses.color_content, curses.COLORS + 1) + self.assertRaises(ValueError, curses.color_content, -2**31 - 1) + self.assertRaises(ValueError, curses.color_content, 2**31) + self.assertRaises(ValueError, curses.color_content, -2**63 - 1) + self.assertRaises(ValueError, curses.color_content, 2**63 - 1) + self.assertRaises(ValueError, curses.pair_content, -1) + self.assertRaises(ValueError, curses.pair_content, curses.COLOR_PAIRS) + self.assertRaises(ValueError, curses.pair_content, -2**31 - 1) + self.assertRaises(ValueError, curses.pair_content, 2**31) + self.assertRaises(ValueError, curses.pair_content, -2**63 - 1) + self.assertRaises(ValueError, curses.pair_content, 2**63 - 1) + @requires_curses_func('keyname') def test_keyname(self): curses.keyname(13) diff --git a/Misc/ACKS b/Misc/ACKS index f5e9459276c86..1599b09c692b7 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -798,6 +798,7 @@ Geert Jansen Jack Jansen Hans-Peter Jansen Bill Janssen +Hans Petter Jansson Jon Janzen Thomas Jarosch Juhana Jauhiainen @@ -882,6 +883,7 @@ Sam Kimbrel Tomohiko Kinebuchi James King W. Trevor King +Jeffrey Kintscher Paul Kippes Steve Kirsch Sebastian Kirsche diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-05-25-05-27-39.bpo-36982.0UHgfB.rst b/Misc/NEWS.d/next/Core and Builtins/2019-05-25-05-27-39.bpo-36982.0UHgfB.rst new file mode 100644 index 0000000000000..f105f1857d487 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-05-25-05-27-39.bpo-36982.0UHgfB.rst @@ -0,0 +1 @@ +Use ncurses extended color functions when available to support terminals with 256 colors, and add the new function :func:`curses.has_extended_color_support` to indicate whether extended color support is provided by the underlying ncurses library. \ No newline at end of file diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index c70b0e2a19fad..34331017f85c7 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -134,6 +134,31 @@ typedef chtype attr_t; /* No attr_t type is available */ #define STRICT_SYSV_CURSES #endif +#if defined(NCURSES_EXT_COLORS) && defined(NCURSES_EXT_FUNCS) +#define _NCURSES_EXTENDED_COLOR_FUNCS 1 +#else +#define _NCURSES_EXTENDED_COLOR_FUNCS 0 +#endif /* defined(NCURSES_EXT_COLORS) && defined(NCURSES_EXT_FUNCS) */ + +#if _NCURSES_EXTENDED_COLOR_FUNCS +#define _NCURSES_COLOR_VAL_TYPE int +#define _CURSES_INIT_COLOR_FUNC init_extended_color +#define _CURSES_INIT_PAIR_FUNC init_extended_pair +#define _COLOR_CONTENT_FUNC extended_color_content +#define _CURSES_PAIR_NUMBER_FUNC extended_pair_content +#else +#define _NCURSES_COLOR_VAL_TYPE short +#define _CURSES_INIT_COLOR_FUNC init_color +#define _CURSES_INIT_PAIR_FUNC init_pair +#define _COLOR_CONTENT_FUNC color_content +#define _CURSES_PAIR_NUMBER_FUNC pair_content +#endif /* _NCURSES_EXTENDED_COLOR_FUNCS */ + +#define _CURSES_FUNC_NAME_STR(s) #s + +#define _CURSES_INIT_COLOR_FUNC_NAME _CURSES_FUNC_NAME_STR(_CURSES_INIT_COLOR_FUNC) +#define _CURSES_INIT_PAIR_FUNC_NAME _CURSES_FUNC_NAME_STR(_CURSES_INIT_PAIR_FUNC) + /*[clinic input] module _curses class _curses.window "PyCursesWindowObject *" "&PyCursesWindow_Type" @@ -387,6 +412,104 @@ PyCurses_ConvertToString(PyCursesWindowObject *win, PyObject *obj, return 0; } +static int +color_converter(PyObject *arg, void *ptr) +{ + long color_number; + int overflow; + + color_number = PyLong_AsLongAndOverflow(arg, &overflow); + if (color_number == -1 && PyErr_Occurred()) + return 0; + + if (overflow > 0 || color_number > COLORS) { + PyErr_Format(PyExc_ValueError, + "Color number is greater than COLORS (%d).", + COLORS); + return 0; + } + else if (overflow < 0 || color_number < 0) { + PyErr_SetString(PyExc_ValueError, + "Color number is less than 0."); + return 0; + } + + *(int *)ptr = (int)color_number; + return 1; +} + +/*[python input] +class color_converter(CConverter): + type = 'int' + converter = 'color_converter' +[python start generated code]*/ +/*[python end generated code: output=da39a3ee5e6b4b0d input=4260d2b6e66b3709]*/ + +static int +pair_converter(PyObject *arg, void *ptr) +{ + long pair_number; + int overflow; + + pair_number = PyLong_AsLongAndOverflow(arg, &overflow); + if (pair_number == -1 && PyErr_Occurred()) + return 0; + + if (overflow > 0 || pair_number > COLOR_PAIRS - 1) { + PyErr_Format(PyExc_ValueError, + "Color pair is greater than COLOR_PAIRS-1 (%d).", + COLOR_PAIRS - 1); + return 0; + } + else if (overflow < 0 || pair_number < 1) { + PyErr_SetString(PyExc_ValueError, + "Color pair is less than 1."); + return 0; + } + + *(int *)ptr = (int)pair_number; + return 1; +} + +/*[python input] +class pair_converter(CConverter): + type = 'int' + converter = 'pair_converter' +[python start generated code]*/ +/*[python end generated code: output=da39a3ee5e6b4b0d input=1a918ae6a1b32af7]*/ + +static int +component_converter(PyObject *arg, void *ptr) +{ + long component; + int overflow; + + component = PyLong_AsLongAndOverflow(arg, &overflow); + if (component == -1 && PyErr_Occurred()) + return 0; + + if (overflow > 0 || component > 1000) { + PyErr_SetString(PyExc_ValueError, + "Color component is greater than 1000"); + return 0; + } + else if (overflow < 0 || component < 0) { + PyErr_SetString(PyExc_ValueError, + "Color component is less than 0"); + return 0; + } + + *(short *)ptr = (short)component; + return 1; +} + +/*[python input] +class component_converter(CConverter): + type = 'short' + converter = 'component_converter' +[python start generated code]*/ +/*[python end generated code: output=da39a3ee5e6b4b0d input=38e9be01d33927fb]*/ + /* Function versions of the 3 functions for testing whether curses has been initialised or not. */ @@ -2585,7 +2708,7 @@ NoArgOrFlagNoReturnFunctionBody(cbreak, flag) /*[clinic input] _curses.color_content - color_number: short + color_number: color The number of the color (0 - COLORS). / @@ -2596,15 +2719,15 @@ which will be between 0 (no component) and 1000 (maximum amount of component). [clinic start generated code]*/ static PyObject * -_curses_color_content_impl(PyObject *module, short color_number) -/*[clinic end generated code: output=cb15cf3120d4bfc1 input=5555abb1c11e11b7]*/ +_curses_color_content_impl(PyObject *module, int color_number) +/*[clinic end generated code: output=17b466df7054e0de input=c10ef58f694b13ee]*/ { - short r,g,b; + _NCURSES_COLOR_VAL_TYPE r,g,b; PyCursesInitialised; PyCursesInitialisedColor; - if (color_content(color_number, &r, &g, &b) != ERR) + if (_COLOR_CONTENT_FUNC(color_number, &r, &g, &b) != ERR) return Py_BuildValue("(iii)", r, g, b); else { PyErr_SetString(PyCursesError, @@ -2616,7 +2739,7 @@ _curses_color_content_impl(PyObject *module, short color_number) /*[clinic input] _curses.color_pair - color_number: short + color_number: color The number of the color (0 - COLORS). / @@ -2627,8 +2750,8 @@ other A_* attributes. pair_number() is the counterpart to this function. [clinic start generated code]*/ static PyObject * -_curses_color_pair_impl(PyObject *module, short color_number) -/*[clinic end generated code: output=6a84cb6b29ecaf9a input=a9d3eb6f50e4dc12]*/ +_curses_color_pair_impl(PyObject *module, int color_number) +/*[clinic end generated code: output=3fd752e8e24c93fb input=b049033819ab4ef5]*/ { PyCursesInitialised; PyCursesInitialisedColor; @@ -3027,13 +3150,13 @@ _curses_has_key_impl(PyObject *module, int key) /*[clinic input] _curses.init_color - color_number: short + color_number: color The number of the color to be changed (0 - COLORS). - r: short + r: component Red component (0 - 1000). - g: short + g: component Green component (0 - 1000). - b: short + b: component Blue component (0 - 1000). / @@ -3045,24 +3168,24 @@ most terminals; it is active only if can_change_color() returns 1. [clinic start generated code]*/ static PyObject * -_curses_init_color_impl(PyObject *module, short color_number, short r, - short g, short b) -/*[clinic end generated code: output=280236f5efe9776a input=f3a05bd38f619175]*/ +_curses_init_color_impl(PyObject *module, int color_number, short r, short g, + short b) +/*[clinic end generated code: output=d7ed71b2d818cdf2 input=8a2fe94ca9204aa5]*/ { PyCursesInitialised; PyCursesInitialisedColor; - return PyCursesCheckERR(init_color(color_number, r, g, b), "init_color"); + return PyCursesCheckERR(_CURSES_INIT_COLOR_FUNC(color_number, r, g, b), _CURSES_INIT_COLOR_FUNC_NAME); } /*[clinic input] _curses.init_pair - pair_number: short + pair_number: pair The number of the color-pair to be changed (1 - (COLOR_PAIRS-1)). - fg: short + fg: color Foreground color number (0 - COLORS). - bg: short + bg: color Background color number (0 - COLORS). / @@ -3073,14 +3196,13 @@ all occurrences of that color-pair are changed to the new definition. [clinic start generated code]*/ static PyObject * -_curses_init_pair_impl(PyObject *module, short pair_number, short fg, - short bg) -/*[clinic end generated code: output=9c2ce39c22f376b6 input=c9f0b11b17a2ac6d]*/ +_curses_init_pair_impl(PyObject *module, int pair_number, int fg, int bg) +/*[clinic end generated code: output=a0bba03d2bbc3ee6 input=b865583a18061c1f]*/ { PyCursesInitialised; PyCursesInitialisedColor; - return PyCursesCheckERR(init_pair(pair_number, fg, bg), "init_pair"); + return PyCursesCheckERR(_CURSES_INIT_PAIR_FUNC(pair_number, fg, bg), _CURSES_INIT_PAIR_FUNC_NAME); } static PyObject *ModDict; @@ -3697,7 +3819,7 @@ NoArgNoReturnFunctionBody(noraw) /*[clinic input] _curses.pair_content - pair_number: short + pair_number: pair The number of the color pair (1 - (COLOR_PAIRS-1)). / @@ -3705,15 +3827,15 @@ Return a tuple (fg, bg) containing the colors for the requested color pair. [clinic start generated code]*/ static PyObject * -_curses_pair_content_impl(PyObject *module, short pair_number) -/*[clinic end generated code: output=5a72aa1a28bbacf3 input=f4d7fec5643b976b]*/ +_curses_pair_content_impl(PyObject *module, int pair_number) +/*[clinic end generated code: output=4a726dd0e6885f3f input=b42eacf8a4103852]*/ { - short f, b; + _NCURSES_COLOR_VAL_TYPE f, b; PyCursesInitialised; PyCursesInitialisedColor; - if (pair_content(pair_number, &f, &b)==ERR) { + if (_CURSES_PAIR_NUMBER_FUNC(pair_number, &f, &b)==ERR) { PyErr_SetString(PyCursesError, "Argument 1 was out of range. (1..COLOR_PAIRS-1)"); return NULL; @@ -4450,6 +4572,21 @@ make_ncurses_version(void) #endif /* NCURSES_VERSION */ +/*[clinic input] +_curses.has_extended_color_support + +Return True if the module supports extended colors; otherwise, return False. + +Extended color support allows more than 256 color-pairs for terminals +that support more than 16 colors (e.g. xterm-256color). +[clinic start generated code]*/ + +static PyObject * +_curses_has_extended_color_support_impl(PyObject *module) +/*[clinic end generated code: output=68f1be2b57d92e22 input=4b905f046e35ee9f]*/ +{ + return PyBool_FromLong(_NCURSES_EXTENDED_COLOR_FUNCS); +} /* List of functions defined in the module */ @@ -4476,6 +4613,7 @@ static PyMethodDef PyCurses_methods[] = { _CURSES_GETSYX_METHODDEF _CURSES_GETWIN_METHODDEF _CURSES_HAS_COLORS_METHODDEF + _CURSES_HAS_EXTENDED_COLOR_SUPPORT_METHODDEF _CURSES_HAS_IC_METHODDEF _CURSES_HAS_IL_METHODDEF _CURSES_HAS_KEY_METHODDEF diff --git a/Modules/clinic/_cursesmodule.c.h b/Modules/clinic/_cursesmodule.c.h index f686ded51976c..c4c2b71e4cc22 100644 --- a/Modules/clinic/_cursesmodule.c.h +++ b/Modules/clinic/_cursesmodule.c.h @@ -1967,32 +1967,16 @@ PyDoc_STRVAR(_curses_color_content__doc__, {"color_content", (PyCFunction)_curses_color_content, METH_O, _curses_color_content__doc__}, static PyObject * -_curses_color_content_impl(PyObject *module, short color_number); +_curses_color_content_impl(PyObject *module, int color_number); static PyObject * _curses_color_content(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; - short color_number; + int color_number; - { - long ival = PyLong_AsLong(arg); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - color_number = (short) ival; - } + if (!color_converter(arg, &color_number)) { + goto exit; } return_value = _curses_color_content_impl(module, color_number); @@ -2016,32 +2000,16 @@ PyDoc_STRVAR(_curses_color_pair__doc__, {"color_pair", (PyCFunction)_curses_color_pair, METH_O, _curses_color_pair__doc__}, static PyObject * -_curses_color_pair_impl(PyObject *module, short color_number); +_curses_color_pair_impl(PyObject *module, int color_number); static PyObject * _curses_color_pair(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; - short color_number; + int color_number; - { - long ival = PyLong_AsLong(arg); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - color_number = (short) ival; - } + if (!color_converter(arg, &color_number)) { + goto exit; } return_value = _curses_color_pair_impl(module, color_number); @@ -2590,14 +2558,14 @@ PyDoc_STRVAR(_curses_init_color__doc__, {"init_color", (PyCFunction)(void(*)(void))_curses_init_color, METH_FASTCALL, _curses_init_color__doc__}, static PyObject * -_curses_init_color_impl(PyObject *module, short color_number, short r, - short g, short b); +_curses_init_color_impl(PyObject *module, int color_number, short r, short g, + short b); static PyObject * _curses_init_color(PyObject *module, PyObject *const *args, Py_ssize_t nargs) { PyObject *return_value = NULL; - short color_number; + int color_number; short r; short g; short b; @@ -2605,81 +2573,17 @@ _curses_init_color(PyObject *module, PyObject *const *args, Py_ssize_t nargs) if (!_PyArg_CheckPositional("init_color", nargs, 4, 4)) { goto exit; } - { - long ival = PyLong_AsLong(args[0]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - color_number = (short) ival; - } + if (!color_converter(args[0], &color_number)) { + goto exit; } - { - long ival = PyLong_AsLong(args[1]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - r = (short) ival; - } + if (!component_converter(args[1], &r)) { + goto exit; } - { - long ival = PyLong_AsLong(args[2]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - g = (short) ival; - } + if (!component_converter(args[2], &g)) { + goto exit; } - { - long ival = PyLong_AsLong(args[3]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - b = (short) ival; - } + if (!component_converter(args[3], &b)) { + goto exit; } return_value = _curses_init_color_impl(module, color_number, r, g, b); @@ -2707,76 +2611,27 @@ PyDoc_STRVAR(_curses_init_pair__doc__, {"init_pair", (PyCFunction)(void(*)(void))_curses_init_pair, METH_FASTCALL, _curses_init_pair__doc__}, static PyObject * -_curses_init_pair_impl(PyObject *module, short pair_number, short fg, - short bg); +_curses_init_pair_impl(PyObject *module, int pair_number, int fg, int bg); static PyObject * _curses_init_pair(PyObject *module, PyObject *const *args, Py_ssize_t nargs) { PyObject *return_value = NULL; - short pair_number; - short fg; - short bg; + int pair_number; + int fg; + int bg; if (!_PyArg_CheckPositional("init_pair", nargs, 3, 3)) { goto exit; } - { - long ival = PyLong_AsLong(args[0]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - pair_number = (short) ival; - } + if (!pair_converter(args[0], &pair_number)) { + goto exit; } - { - long ival = PyLong_AsLong(args[1]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - fg = (short) ival; - } + if (!color_converter(args[1], &fg)) { + goto exit; } - { - long ival = PyLong_AsLong(args[2]); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - bg = (short) ival; - } + if (!color_converter(args[2], &bg)) { + goto exit; } return_value = _curses_init_pair_impl(module, pair_number, fg, bg); @@ -3554,32 +3409,16 @@ PyDoc_STRVAR(_curses_pair_content__doc__, {"pair_content", (PyCFunction)_curses_pair_content, METH_O, _curses_pair_content__doc__}, static PyObject * -_curses_pair_content_impl(PyObject *module, short pair_number); +_curses_pair_content_impl(PyObject *module, int pair_number); static PyObject * _curses_pair_content(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; - short pair_number; + int pair_number; - { - long ival = PyLong_AsLong(arg); - if (ival == -1 && PyErr_Occurred()) { - goto exit; - } - else if (ival < SHRT_MIN) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is less than minimum"); - goto exit; - } - else if (ival > SHRT_MAX) { - PyErr_SetString(PyExc_OverflowError, - "signed short integer is greater than maximum"); - goto exit; - } - else { - pair_number = (short) ival; - } + if (!pair_converter(arg, &pair_number)) { + goto exit; } return_value = _curses_pair_content_impl(module, pair_number); @@ -4337,6 +4176,27 @@ _curses_use_default_colors(PyObject *module, PyObject *Py_UNUSED(ignored)) #endif /* !defined(STRICT_SYSV_CURSES) */ +PyDoc_STRVAR(_curses_has_extended_color_support__doc__, +"has_extended_color_support($module, /)\n" +"--\n" +"\n" +"Return True if the module supports extended colors; otherwise, return False.\n" +"\n" +"Extended color support allows more than 256 color-pairs for terminals\n" +"that support more than 16 colors (e.g. xterm-256color)."); + +#define _CURSES_HAS_EXTENDED_COLOR_SUPPORT_METHODDEF \ + {"has_extended_color_support", (PyCFunction)_curses_has_extended_color_support, METH_NOARGS, _curses_has_extended_color_support__doc__}, + +static PyObject * +_curses_has_extended_color_support_impl(PyObject *module); + +static PyObject * +_curses_has_extended_color_support(PyObject *module, PyObject *Py_UNUSED(ignored)) +{ + return _curses_has_extended_color_support_impl(module); +} + #ifndef _CURSES_WINDOW_ENCLOSE_METHODDEF #define _CURSES_WINDOW_ENCLOSE_METHODDEF #endif /* !defined(_CURSES_WINDOW_ENCLOSE_METHODDEF) */ @@ -4428,4 +4288,4 @@ _curses_use_default_colors(PyObject *module, PyObject *Py_UNUSED(ignored)) #ifndef _CURSES_USE_DEFAULT_COLORS_METHODDEF #define _CURSES_USE_DEFAULT_COLORS_METHODDEF #endif /* !defined(_CURSES_USE_DEFAULT_COLORS_METHODDEF) */ -/*[clinic end generated code: output=478d93f7692385eb input=a9049054013a1b77]*/ +/*[clinic end generated code: output=38b2531d17f119e1 input=a9049054013a1b77]*/ From webhook-mailer at python.org Tue Aug 4 11:52:08 2020 From: webhook-mailer at python.org (Hai Shi) Date: Tue, 04 Aug 2020 15:52:08 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21452) Message-ID: https://github.com/python/cpython/commit/604bba1f8fe32479c89b9824b2231cc4480dd110 commit: 604bba1f8fe32479c89b9824b2231cc4480dd110 branch: master author: Hai Shi committer: GitHub date: 2020-08-04T17:51:43+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21452) files: M Lib/test/eintrdata/eintr_tester.py M Lib/test/test_email/test_email.py M Lib/test/test_multibytecodec.py M Lib/test/test_pdb.py M Lib/test/test_urllib2_localnet.py M Lib/test/test_warnings/__init__.py diff --git a/Lib/test/eintrdata/eintr_tester.py b/Lib/test/eintrdata/eintr_tester.py index 606f31b091096..e43b59d064f55 100644 --- a/Lib/test/eintrdata/eintr_tester.py +++ b/Lib/test/eintrdata/eintr_tester.py @@ -22,6 +22,7 @@ import unittest from test import support +from test.support import os_helper from test.support import socket_helper @contextlib.contextmanager @@ -314,16 +315,16 @@ def test_accept(self): @support.requires_freebsd_version(10, 3) @unittest.skipUnless(hasattr(os, 'mkfifo'), 'needs mkfifo()') def _test_open(self, do_open_close_reader, do_open_close_writer): - filename = support.TESTFN + filename = os_helper.TESTFN # Use a fifo: until the child opens it for reading, the parent will # block when trying to open it for writing. - support.unlink(filename) + os_helper.unlink(filename) try: os.mkfifo(filename) except PermissionError as e: self.skipTest('os.mkfifo(): %s' % e) - self.addCleanup(support.unlink, filename) + self.addCleanup(os_helper.unlink, filename) code = '\n'.join(( 'import os, time', @@ -486,16 +487,16 @@ def test_devpoll(self): class FNTLEINTRTest(EINTRBaseTest): def _lock(self, lock_func, lock_name): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) code = '\n'.join(( "import fcntl, time", - "with open('%s', 'wb') as f:" % support.TESTFN, + "with open('%s', 'wb') as f:" % os_helper.TESTFN, " fcntl.%s(f, fcntl.LOCK_EX)" % lock_name, " time.sleep(%s)" % self.sleep_time)) start_time = time.monotonic() proc = self.subprocess(code) with kill_on_error(proc): - with open(support.TESTFN, 'wb') as f: + with open(os_helper.TESTFN, 'wb') as f: while True: # synchronize the subprocess dt = time.monotonic() - start_time if dt > 60.0: diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 1d28e26dec681..ba4ed69cc9b34 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -38,7 +38,7 @@ from email import quoprimime from test.support import threading_helper -from test.support import unlink +from test.support.os_helper import unlink from test.test_email import openfile, TestEmailBase # These imports are documented to work, but we are testing them using a diff --git a/Lib/test/test_multibytecodec.py b/Lib/test/test_multibytecodec.py index 3cf5d7beb144b..7c3b67f3cbf6e 100644 --- a/Lib/test/test_multibytecodec.py +++ b/Lib/test/test_multibytecodec.py @@ -4,7 +4,8 @@ # from test import support -from test.support import TESTFN +from test.support import os_helper +from test.support.os_helper import TESTFN import unittest, io, codecs, sys import _multibytecodec @@ -57,7 +58,7 @@ def test_codingspec(self): code = '# coding: {}\n'.format(enc) exec(code) finally: - support.unlink(TESTFN) + os_helper.unlink(TESTFN) def test_init_segfault(self): # bug #3305: this used to segfault @@ -296,7 +297,7 @@ def test_bug1728403(self): finally: f.close() finally: - support.unlink(TESTFN) + os_helper.unlink(TESTFN) class Test_StreamWriter(unittest.TestCase): def test_gb18030(self): diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index 65bca291d9618..1a2bbb382e864 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -12,7 +12,7 @@ from contextlib import ExitStack from io import StringIO -from test import support +from test.support import os_helper # This little helper class is essential for testing pdb under doctest. from test.test_doctest import _FakeInput from unittest.mock import patch @@ -1188,10 +1188,10 @@ def test_pdb_issue_20766(): class PdbTestCase(unittest.TestCase): def tearDown(self): - support.unlink(support.TESTFN) + os_helper.unlink(os_helper.TESTFN) def _run_pdb(self, pdb_args, commands): - self.addCleanup(support.rmtree, '__pycache__') + self.addCleanup(os_helper.rmtree, '__pycache__') cmd = [sys.executable, '-m', 'pdb'] + pdb_args with subprocess.Popen( cmd, @@ -1210,13 +1210,13 @@ def run_pdb_script(self, script, commands): filename = 'main.py' with open(filename, 'w') as f: f.write(textwrap.dedent(script)) - self.addCleanup(support.unlink, filename) + self.addCleanup(os_helper.unlink, filename) return self._run_pdb([filename], commands) def run_pdb_module(self, script, commands): """Runs the script code as part of a module""" self.module_name = 't_main' - support.rmtree(self.module_name) + os_helper.rmtree(self.module_name) main_file = self.module_name + '/__main__.py' init_file = self.module_name + '/__init__.py' os.mkdir(self.module_name) @@ -1224,17 +1224,17 @@ def run_pdb_module(self, script, commands): pass with open(main_file, 'w') as f: f.write(textwrap.dedent(script)) - self.addCleanup(support.rmtree, self.module_name) + self.addCleanup(os_helper.rmtree, self.module_name) return self._run_pdb(['-m', self.module_name], commands) def _assert_find_function(self, file_content, func_name, expected): - with open(support.TESTFN, 'wb') as f: + with open(os_helper.TESTFN, 'wb') as f: f.write(file_content) expected = None if not expected else ( - expected[0], support.TESTFN, expected[1]) + expected[0], os_helper.TESTFN, expected[1]) self.assertEqual( - expected, pdb.find_function(func_name, support.TESTFN)) + expected, pdb.find_function(func_name, os_helper.TESTFN)) def test_find_function_empty_file(self): self._assert_find_function(b'', 'foo', None) @@ -1284,9 +1284,9 @@ def b?r(): def test_issue7964(self): # open the file as binary so we can force \r\n newline - with open(support.TESTFN, 'wb') as f: + with open(os_helper.TESTFN, 'wb') as f: f.write(b'print("testing my pdb")\r\n') - cmd = [sys.executable, '-m', 'pdb', support.TESTFN] + cmd = [sys.executable, '-m', 'pdb', os_helper.TESTFN] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, @@ -1327,7 +1327,7 @@ def bar(): """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) - self.addCleanup(support.unlink, 'bar.py') + self.addCleanup(os_helper.unlink, 'bar.py') stdout, stderr = self.run_pdb_script(script, commands) self.assertTrue( any('main.py(5)foo()->None' in l for l in stdout.splitlines()), @@ -1337,7 +1337,7 @@ def test_issue13120(self): # Invoking "continue" on a non-main thread triggered an exception # inside signal.signal. - with open(support.TESTFN, 'wb') as f: + with open(os_helper.TESTFN, 'wb') as f: f.write(textwrap.dedent(""" import threading import pdb @@ -1349,7 +1349,7 @@ def start_pdb(): t = threading.Thread(target=start_pdb) t.start()""").encode('ascii')) - cmd = [sys.executable, '-u', support.TESTFN] + cmd = [sys.executable, '-u', os_helper.TESTFN] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, @@ -1363,7 +1363,7 @@ def start_pdb(): def test_issue36250(self): - with open(support.TESTFN, 'wb') as f: + with open(os_helper.TESTFN, 'wb') as f: f.write(textwrap.dedent(""" import threading import pdb @@ -1379,7 +1379,7 @@ def start_pdb(): pdb.Pdb(readrc=False).set_trace() evt.set() t.join()""").encode('ascii')) - cmd = [sys.executable, '-u', support.TESTFN] + cmd = [sys.executable, '-u', os_helper.TESTFN] proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, @@ -1412,7 +1412,7 @@ def test_readrc_kwarg(self): save_home = os.environ.pop('HOME', None) try: - with support.temp_cwd(): + with os_helper.temp_cwd(): with open('.pdbrc', 'w') as f: f.write("invalid\n") @@ -1437,7 +1437,7 @@ def test_readrc_kwarg(self): def test_readrc_homedir(self): save_home = os.environ.pop("HOME", None) - with support.temp_dir() as temp_dir, patch("os.path.expanduser"): + with os_helper.temp_dir() as temp_dir, patch("os.path.expanduser"): rc_path = os.path.join(temp_dir, ".pdbrc") os.path.expanduser.return_value = rc_path try: @@ -1506,12 +1506,12 @@ def test_run_pdb_with_pdb(self): def test_module_without_a_main(self): module_name = 't_main' - support.rmtree(module_name) + os_helper.rmtree(module_name) init_file = module_name + '/__init__.py' os.mkdir(module_name) with open(init_file, 'w') as f: pass - self.addCleanup(support.rmtree, module_name) + self.addCleanup(os_helper.rmtree, module_name) stdout, stderr = self._run_pdb(['-m', module_name], "") self.assertIn("ImportError: No module named t_main.__main__", stdout.splitlines()) @@ -1531,11 +1531,11 @@ def test_blocks_at_first_code_line(self): def test_relative_imports(self): self.module_name = 't_main' - support.rmtree(self.module_name) + os_helper.rmtree(self.module_name) main_file = self.module_name + '/__main__.py' init_file = self.module_name + '/__init__.py' module_file = self.module_name + '/module.py' - self.addCleanup(support.rmtree, self.module_name) + self.addCleanup(os_helper.rmtree, self.module_name) os.mkdir(self.module_name) with open(init_file, 'w') as f: f.write(textwrap.dedent(""" @@ -1569,11 +1569,11 @@ def test_relative_imports(self): def test_relative_imports_on_plain_module(self): # Validates running a plain module. See bpo32691 self.module_name = 't_main' - support.rmtree(self.module_name) + os_helper.rmtree(self.module_name) main_file = self.module_name + '/runme.py' init_file = self.module_name + '/__init__.py' module_file = self.module_name + '/module.py' - self.addCleanup(support.rmtree, self.module_name) + self.addCleanup(os_helper.rmtree, self.module_name) os.mkdir(self.module_name) with open(init_file, 'w') as f: f.write(textwrap.dedent(""" diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py index e568cc4575549..ebb43c30b4d50 100644 --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -8,9 +8,9 @@ import unittest import hashlib -from test import support from test.support import hashlib_helper from test.support import threading_helper +from test.support import warnings_helper try: import ssl @@ -567,7 +567,7 @@ def test_https(self): def test_https_with_cafile(self): handler = self.start_https_server(certfile=CERT_localhost) - with support.check_warnings(('', DeprecationWarning)): + with warnings_helper.check_warnings(('', DeprecationWarning)): # Good cert data = self.urlopen("https://localhost:%s/bizarre" % handler.port, cafile=CERT_localhost) @@ -585,7 +585,7 @@ def test_https_with_cafile(self): def test_https_with_cadefault(self): handler = self.start_https_server(certfile=CERT_localhost) # Self-signed cert should fail verification with system certificate store - with support.check_warnings(('', DeprecationWarning)): + with warnings_helper.check_warnings(('', DeprecationWarning)): with self.assertRaises(urllib.error.URLError) as cm: self.urlopen("https://localhost:%s/bizarre" % handler.port, cadefault=True) diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index dcc0ea89b5a85..04f7560ecc09c 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -7,14 +7,20 @@ import textwrap import unittest from test import support +from test.support import import_helper +from test.support import os_helper +from test.support import warnings_helper from test.support.script_helper import assert_python_ok, assert_python_failure from test.test_warnings.data import stacklevel as warning_tests import warnings as original_warnings -py_warnings = support.import_fresh_module('warnings', blocked=['_warnings']) -c_warnings = support.import_fresh_module('warnings', fresh=['_warnings']) + +py_warnings = import_helper.import_fresh_module('warnings', + blocked=['_warnings']) +c_warnings = import_helper.import_fresh_module('warnings', + fresh=['_warnings']) Py_DEBUG = hasattr(sys, 'gettotalrefcount') @@ -440,7 +446,7 @@ def test_stacklevel(self): def test_stacklevel_import(self): # Issue #24305: With stacklevel=2, module-level warnings should work. - support.unload('test.test_warnings.data.import_warning') + import_helper.unload('test.test_warnings.data.import_warning') with warnings_state(self.module): with original_warnings.catch_warnings(record=True, module=self.module) as w: @@ -543,7 +549,7 @@ class CWarnTests(WarnTests, unittest.TestCase): module = c_warnings # As an early adopter, we sanity check the - # test.support.import_fresh_module utility function + # test.import_helper.import_fresh_module utility function def test_accelerated(self): self.assertIsNot(original_warnings, self.module) self.assertFalse(hasattr(self.module.warn, '__code__')) @@ -552,7 +558,7 @@ class PyWarnTests(WarnTests, unittest.TestCase): module = py_warnings # As an early adopter, we sanity check the - # test.support.import_fresh_module utility function + # test.import_helper.import_fresh_module utility function def test_pure_python(self): self.assertIsNot(original_warnings, self.module) self.assertTrue(hasattr(self.module.warn, '__code__')) @@ -927,9 +933,9 @@ class PyWarningsDisplayTests(WarningsDisplayTests, unittest.TestCase): module = py_warnings def test_tracemalloc(self): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) - with open(support.TESTFN, 'w') as fp: + with open(os_helper.TESTFN, 'w') as fp: fp.write(textwrap.dedent(""" def func(): f = open(__file__) @@ -949,8 +955,8 @@ def run(*args): return stderr # tracemalloc disabled - filename = os.path.abspath(support.TESTFN) - stderr = run('-Wd', support.TESTFN) + filename = os.path.abspath(os_helper.TESTFN) + stderr = run('-Wd', os_helper.TESTFN) expected = textwrap.dedent(f''' {filename}:5: ResourceWarning: unclosed file <...> f = None @@ -959,7 +965,7 @@ def run(*args): self.assertEqual(stderr, expected) # tracemalloc enabled - stderr = run('-Wd', '-X', 'tracemalloc=2', support.TESTFN) + stderr = run('-Wd', '-X', 'tracemalloc=2', os_helper.TESTFN) expected = textwrap.dedent(f''' {filename}:5: ResourceWarning: unclosed file <...> f = None @@ -1093,7 +1099,7 @@ def test_check_warnings(self): wmod = self.module if wmod is not sys.modules['warnings']: self.skipTest('module to test is not loaded warnings module') - with support.check_warnings(quiet=False) as w: + with warnings_helper.check_warnings(quiet=False) as w: self.assertEqual(w.warnings, []) wmod.simplefilter("always") wmod.warn("foo") @@ -1105,18 +1111,18 @@ def test_check_warnings(self): w.reset() self.assertEqual(w.warnings, []) - with support.check_warnings(): + with warnings_helper.check_warnings(): # defaults to quiet=True without argument pass - with support.check_warnings(('foo', UserWarning)): + with warnings_helper.check_warnings(('foo', UserWarning)): wmod.warn("foo") with self.assertRaises(AssertionError): - with support.check_warnings(('', RuntimeWarning)): + with warnings_helper.check_warnings(('', RuntimeWarning)): # defaults to quiet=False with argument pass with self.assertRaises(AssertionError): - with support.check_warnings(('foo', RuntimeWarning)): + with warnings_helper.check_warnings(('foo', RuntimeWarning)): wmod.warn("foo") class CCatchWarningTests(CatchWarningTests, unittest.TestCase): @@ -1198,7 +1204,7 @@ def test_default_filter_configuration(self): @unittest.skipUnless(sys.getfilesystemencoding() != 'ascii', 'requires non-ascii filesystemencoding') def test_nonascii(self): - PYTHONWARNINGS="ignore:DeprecationWarning" + support.FS_NONASCII + PYTHONWARNINGS="ignore:DeprecationWarning" + os_helper.FS_NONASCII rc, stdout, stderr = assert_python_ok("-c", "import sys; sys.stdout.write(str(sys.warnoptions))", PYTHONIOENCODING="utf-8", @@ -1218,7 +1224,7 @@ def test_issue_8766(self): # "import encodings" emits a warning whereas the warnings is not loaded # or not completely loaded (warnings imports indirectly encodings by # importing linecache) yet - with support.temp_cwd() as cwd, support.temp_cwd('encodings'): + with os_helper.temp_cwd() as cwd, os_helper.temp_cwd('encodings'): # encodings loaded by initfsencoding() assert_python_ok('-c', 'pass', PYTHONPATH=cwd) From webhook-mailer at python.org Tue Aug 4 11:53:17 2020 From: webhook-mailer at python.org (Hai Shi) Date: Tue, 04 Aug 2020 15:53:17 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21727) Message-ID: https://github.com/python/cpython/commit/c7decc27d529c04a4e6b2922e3f3f9419b920f63 commit: c7decc27d529c04a4e6b2922e3f3f9419b920f63 branch: master author: Hai Shi committer: GitHub date: 2020-08-04T17:53:12+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21727) files: M Lib/test/test_importlib/fixtures.py M Lib/test/test_importlib/import_/test_packages.py M Lib/test/test_importlib/source/test_file_loader.py M Lib/test/test_importlib/source/test_finder.py M Lib/test/test_importlib/test_abc.py M Lib/test/test_importlib/test_api.py M Lib/test/test_importlib/test_pkg_import.py M Lib/test/test_importlib/test_spec.py M Lib/test/test_importlib/test_threaded_import.py M Lib/test/test_importlib/test_windows.py M Lib/test/test_importlib/util.py M Lib/test/test_inspect.py M Lib/test/test_site.py M Lib/test/test_tools/test_fixcid.py M Lib/test/test_tools/test_i18n.py M Lib/test/test_tools/test_lll.py M Lib/test/test_tools/test_pathfix.py M Lib/test/test_tools/test_pindent.py M Lib/test/test_tools/test_sundry.py diff --git a/Lib/test/test_importlib/fixtures.py b/Lib/test/test_importlib/fixtures.py index 2e55d14b9aab9..985277f64615f 100644 --- a/Lib/test/test_importlib/fixtures.py +++ b/Lib/test/test_importlib/fixtures.py @@ -213,11 +213,11 @@ def build_files(file_defs, prefix=pathlib.Path()): class FileBuilder: def unicode_filename(self): try: - import test.support + from test.support import os_helper except ImportError: # outside CPython, hard-code a unicode snowman return '?' - return test.support.FS_NONASCII or \ + return os_helper.FS_NONASCII or \ self.skip("File system does not support non-ascii.") diff --git a/Lib/test/test_importlib/import_/test_packages.py b/Lib/test/test_importlib/import_/test_packages.py index 24396044a5bcf..c73ac63f6eef3 100644 --- a/Lib/test/test_importlib/import_/test_packages.py +++ b/Lib/test/test_importlib/import_/test_packages.py @@ -2,6 +2,7 @@ import sys import unittest from test import support +from test.support import import_helper class ParentModuleTests: @@ -98,7 +99,7 @@ def module_injection(): try: submodule = self.__import__(subname) finally: - support.unload(subname) + import_helper.unload(subname) (Frozen_ParentTests, diff --git a/Lib/test/test_importlib/source/test_file_loader.py b/Lib/test/test_importlib/source/test_file_loader.py index ab44722146e37..cbd1533b67664 100644 --- a/Lib/test/test_importlib/source/test_file_loader.py +++ b/Lib/test/test_importlib/source/test_file_loader.py @@ -17,7 +17,7 @@ import unittest import warnings -from test.support import make_legacy_pyc, unload +from test.support.import_helper import make_legacy_pyc, unload from test.test_py_compile import without_source_date_epoch from test.test_py_compile import SourceDateEpochTestMeta diff --git a/Lib/test/test_importlib/source/test_finder.py b/Lib/test/test_importlib/source/test_finder.py index f372b850dc7c1..e9207deedc75c 100644 --- a/Lib/test/test_importlib/source/test_finder.py +++ b/Lib/test/test_importlib/source/test_finder.py @@ -9,7 +9,7 @@ import stat import sys import tempfile -from test.support import make_legacy_pyc +from test.support.import_helper import make_legacy_pyc import unittest import warnings diff --git a/Lib/test/test_importlib/test_abc.py b/Lib/test/test_importlib/test_abc.py index 9816b35ef829a..605738fae2e37 100644 --- a/Lib/test/test_importlib/test_abc.py +++ b/Lib/test/test_importlib/test_abc.py @@ -3,6 +3,7 @@ import os import sys from test import support +from test.support import import_helper import types import unittest from unittest import mock @@ -579,8 +580,8 @@ class InspectLoaderLoadModuleTests: module_name = 'blah' def setUp(self): - support.unload(self.module_name) - self.addCleanup(support.unload, self.module_name) + import_helper.unload(self.module_name) + self.addCleanup(import_helper.unload, self.module_name) def load(self, loader): spec = self.util.spec_from_loader(self.module_name, loader) diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py index 0fb1346f9eec8..fd60634e09333 100644 --- a/Lib/test/test_importlib/test_api.py +++ b/Lib/test/test_importlib/test_api.py @@ -7,6 +7,8 @@ import os.path import sys from test import support +from test.support import import_helper +from test.support import os_helper import types import unittest import warnings @@ -200,7 +202,7 @@ class ReloadTests: def test_reload_modules(self): for mod in ('tokenize', 'time', 'marshal'): with self.subTest(module=mod): - with support.CleanImport(mod): + with import_helper.CleanImport(mod): module = self.init.import_module(mod) self.init.reload(module) @@ -221,7 +223,7 @@ def code(): self.assertEqual(reloaded.spam, 3) def test_reload_missing_loader(self): - with support.CleanImport('types'): + with import_helper.CleanImport('types'): import types loader = types.__loader__ del types.__loader__ @@ -232,7 +234,7 @@ def test_reload_missing_loader(self): self.assertEqual(reloaded.__loader__.path, loader.path) def test_reload_loader_replaced(self): - with support.CleanImport('types'): + with import_helper.CleanImport('types'): import types types.__loader__ = None self.init.invalidate_caches() @@ -244,9 +246,9 @@ def test_reload_loader_replaced(self): def test_reload_location_changed(self): name = 'spam' - with support.temp_cwd(None) as cwd: + with os_helper.temp_cwd(None) as cwd: with test_util.uncache('spam'): - with support.DirsOnSysPath(cwd): + with import_helper.DirsOnSysPath(cwd): # Start as a plain module. self.init.invalidate_caches() path = os.path.join(cwd, name + '.py') @@ -257,7 +259,7 @@ def test_reload_location_changed(self): '__cached__': cached, '__doc__': None, } - support.create_empty_file(path) + os_helper.create_empty_file(path) module = self.init.import_module(name) ns = vars(module).copy() loader = ns.pop('__loader__') @@ -295,9 +297,9 @@ def test_reload_location_changed(self): def test_reload_namespace_changed(self): name = 'spam' - with support.temp_cwd(None) as cwd: + with os_helper.temp_cwd(None) as cwd: with test_util.uncache('spam'): - with support.DirsOnSysPath(cwd): + with import_helper.DirsOnSysPath(cwd): # Start as a namespace package. self.init.invalidate_caches() bad_path = os.path.join(cwd, name, '__init.py') diff --git a/Lib/test/test_importlib/test_pkg_import.py b/Lib/test/test_importlib/test_pkg_import.py index 6181dcfab280c..36e78afa10bfc 100644 --- a/Lib/test/test_importlib/test_pkg_import.py +++ b/Lib/test/test_importlib/test_pkg_import.py @@ -7,7 +7,7 @@ import unittest from importlib.util import cache_from_source -from test.support import create_empty_file +from test.support.os_helper import create_empty_file class TestImport(unittest.TestCase): diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py index 20dacec8664e1..eed90f29f9286 100644 --- a/Lib/test/test_importlib/test_spec.py +++ b/Lib/test/test_importlib/test_spec.py @@ -6,7 +6,7 @@ import os.path import pathlib -from test.support import CleanImport +from test.support.import_helper import CleanImport import unittest import sys import warnings diff --git a/Lib/test/test_importlib/test_threaded_import.py b/Lib/test/test_importlib/test_threaded_import.py index 06da18ed396d9..c6a6e1715abad 100644 --- a/Lib/test/test_importlib/test_threaded_import.py +++ b/Lib/test/test_importlib/test_threaded_import.py @@ -14,9 +14,9 @@ import threading import unittest from unittest import mock -from test.support import ( - verbose, run_unittest, TESTFN, - forget, unlink, rmtree) +from test.support import (verbose, run_unittest) +from test.support.import_helper import forget +from test.support.os_helper import (TESTFN, unlink, rmtree) from test.support import threading_helper def task(N, done, done_tasks, errors): diff --git a/Lib/test/test_importlib/test_windows.py b/Lib/test/test_importlib/test_windows.py index 005b685cc03cd..8b3f2009a850d 100644 --- a/Lib/test/test_importlib/test_windows.py +++ b/Lib/test/test_importlib/test_windows.py @@ -6,11 +6,12 @@ import sys import unittest from test import support +from test.support import import_helper from distutils.util import get_platform from contextlib import contextmanager from .util import temp_module -support.import_module('winreg', required_on=['win']) +import_helper.import_module('winreg', required_on=['win']) from winreg import ( CreateKey, HKEY_CURRENT_USER, SetValue, REG_SZ, KEY_ALL_ACCESS, diff --git a/Lib/test/test_importlib/util.py b/Lib/test/test_importlib/util.py index 2745c9b7e3c3b..5c0375e04a6ff 100644 --- a/Lib/test/test_importlib/util.py +++ b/Lib/test/test_importlib/util.py @@ -13,6 +13,7 @@ from pathlib import Path, PurePath from test import support from test.support import import_helper +from test.support import os_helper import unittest import sys import tempfile @@ -159,9 +160,9 @@ def uncache(*names): @contextlib.contextmanager def temp_module(name, content='', *, pkg=False): conflicts = [n for n in sys.modules if n.partition('.')[0] == name] - with support.temp_cwd(None) as cwd: + with os_helper.temp_cwd(None) as cwd: with uncache(name, *conflicts): - with support.DirsOnSysPath(cwd): + with import_helper.DirsOnSysPath(cwd): invalidate_caches() location = os.path.join(cwd, name) @@ -397,7 +398,7 @@ def create_modules(*names): state_manager.__exit__(None, None, None) if uncache_manager is not None: uncache_manager.__exit__(None, None, None) - support.rmtree(temp_dir) + os_helper.rmtree(temp_dir) def mock_path_hook(*entries, importer): @@ -573,8 +574,8 @@ def tearDownClass(cls): pass def setUp(self): - modules = support.modules_setup() - self.addCleanup(support.modules_cleanup, *modules) + modules = import_helper.modules_setup() + self.addCleanup(import_helper.modules_cleanup, *modules) class ZipSetup(ZipSetupBase): diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index e3e2be52076c6..6667dc91edbce 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -24,8 +24,10 @@ except ImportError: ThreadPoolExecutor = None -from test.support import run_unittest, TESTFN, DirsOnSysPath, cpython_only +from test.support import run_unittest, cpython_only from test.support import MISSING_C_DOCSTRINGS, ALWAYS_EQ +from test.support.import_helper import DirsOnSysPath +from test.support.os_helper import TESTFN from test.support.script_helper import assert_python_ok, assert_python_failure from test import inspect_fodder as mod from test import inspect_fodder2 as mod2 diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py index ec86c645981b3..5901939725e18 100644 --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -8,8 +8,8 @@ import test.support from test import support from test.support import socket_helper -from test.support import (captured_stderr, TESTFN, EnvironmentVarGuard, - change_cwd) +from test.support import captured_stderr +from test.support.os_helper import TESTFN, EnvironmentVarGuard, change_cwd import builtins import encodings import glob diff --git a/Lib/test/test_tools/test_fixcid.py b/Lib/test/test_tools/test_fixcid.py index bce029b1aac83..3df1368043719 100644 --- a/Lib/test/test_tools/test_fixcid.py +++ b/Lib/test/test_tools/test_fixcid.py @@ -5,6 +5,7 @@ import runpy import sys from test import support +from test.support import os_helper from test.test_tools import skip_if_missing, scriptsdir import unittest @@ -57,15 +58,15 @@ def test_alter_comments(self): ) def test_directory(self): - os.mkdir(support.TESTFN) - self.addCleanup(support.rmtree, support.TESTFN) - c_filename = os.path.join(support.TESTFN, "file.c") + os.mkdir(os_helper.TESTFN) + self.addCleanup(os_helper.rmtree, os_helper.TESTFN) + c_filename = os.path.join(os_helper.TESTFN, "file.c") with open(c_filename, "w") as file: file.write("int xx;\n") - with open(os.path.join(support.TESTFN, "file.py"), "w") as file: + with open(os.path.join(os_helper.TESTFN, "file.py"), "w") as file: file.write("xx = 'unaltered'\n") script = os.path.join(scriptsdir, "fixcid.py") - output = self.run_script(args=(support.TESTFN,)) + output = self.run_script(args=(os_helper.TESTFN,)) self.assertMultiLineEqual(output, "{}:\n" "1\n" @@ -74,10 +75,10 @@ def test_directory(self): ) def run_script(self, input="", *, args=("-",), substfile="xx yy\n"): - substfilename = support.TESTFN + ".subst" + substfilename = os_helper.TESTFN + ".subst" with open(substfilename, "w") as file: file.write(substfile) - self.addCleanup(support.unlink, substfilename) + self.addCleanup(os_helper.unlink, substfilename) argv = ["fixcid.py", "-s", substfilename] + list(args) script = os.path.join(scriptsdir, "fixcid.py") diff --git a/Lib/test/test_tools/test_i18n.py b/Lib/test/test_tools/test_i18n.py index 42e20f8f7716d..8da657907eab8 100644 --- a/Lib/test/test_tools/test_i18n.py +++ b/Lib/test/test_tools/test_i18n.py @@ -7,7 +7,7 @@ from test.support.script_helper import assert_python_ok from test.test_tools import skip_if_missing, toolsdir -from test.support import temp_cwd, temp_dir +from test.support.os_helper import temp_cwd, temp_dir skip_if_missing() diff --git a/Lib/test/test_tools/test_lll.py b/Lib/test/test_tools/test_lll.py index 568cbfb5e4746..ec0c97334fdeb 100644 --- a/Lib/test/test_tools/test_lll.py +++ b/Lib/test/test_tools/test_lll.py @@ -3,6 +3,7 @@ import os import tempfile from test import support +from test.support import os_helper from test.test_tools import skip_if_missing, import_tool import unittest @@ -14,7 +15,7 @@ class lllTests(unittest.TestCase): def setUp(self): self.lll = import_tool('lll') - @support.skip_unless_symlink + @os_helper.skip_unless_symlink def test_lll_multiple_dirs(self): with tempfile.TemporaryDirectory() as dir1, \ tempfile.TemporaryDirectory() as dir2: diff --git a/Lib/test/test_tools/test_pathfix.py b/Lib/test/test_tools/test_pathfix.py index 03ed29d3f974f..ff61935298b92 100644 --- a/Lib/test/test_tools/test_pathfix.py +++ b/Lib/test/test_tools/test_pathfix.py @@ -3,6 +3,7 @@ import sys import unittest from test import support +from test.support import os_helper from test.test_tools import scriptsdir, skip_if_missing @@ -14,7 +15,7 @@ class TestPathfixFunctional(unittest.TestCase): script = os.path.join(scriptsdir, 'pathfix.py') def setUp(self): - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr='', directory=''): @@ -24,7 +25,7 @@ def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr='', filename = os.path.join(directory, 'script-A_1.py') pathfix_arg = directory else: - filename = support.TESTFN + filename = os_helper.TESTFN pathfix_arg = filename with open(filename, 'w', encoding='utf8') as f: @@ -56,8 +57,8 @@ def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr='', return new_shebang def test_recursive(self): - tmpdir = support.TESTFN + '.d' - self.addCleanup(support.rmtree, tmpdir) + tmpdir = os_helper.TESTFN + '.d' + self.addCleanup(os_helper.rmtree, tmpdir) os.mkdir(tmpdir) expected_stderr = f"recursedown('{os.path.basename(tmpdir)}')\n" self.assertEqual( diff --git a/Lib/test/test_tools/test_pindent.py b/Lib/test/test_tools/test_pindent.py index e293bc872ce51..e7a547ad7d612 100644 --- a/Lib/test/test_tools/test_pindent.py +++ b/Lib/test/test_tools/test_pindent.py @@ -6,6 +6,7 @@ import subprocess import textwrap from test import support +from test.support import os_helper from test.support.script_helper import assert_python_ok from test.test_tools import scriptsdir, skip_if_missing @@ -34,7 +35,7 @@ def lstriplines(self, data): def test_selftest(self): self.maxDiff = None - with support.temp_dir() as directory: + with os_helper.temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: closed = f.read() diff --git a/Lib/test/test_tools/test_sundry.py b/Lib/test/test_tools/test_sundry.py index 10eb6941b3be6..8b5a963e25bd1 100644 --- a/Lib/test/test_tools/test_sundry.py +++ b/Lib/test/test_tools/test_sundry.py @@ -8,7 +8,7 @@ import os import sys import unittest -from test import support +from test.support import import_helper from test.test_tools import scriptsdir, import_tool, skip_if_missing @@ -30,7 +30,7 @@ class TestSundryScripts(unittest.TestCase): skiplist = blacklist + whitelist + windows_only + other def test_sundry(self): - old_modules = support.modules_setup() + old_modules = import_helper.modules_setup() try: for fn in os.listdir(scriptsdir): if not fn.endswith('.py'): @@ -43,7 +43,7 @@ def test_sundry(self): import_tool(name) finally: # Unload all modules loaded in this test - support.modules_cleanup(*old_modules) + import_helper.modules_cleanup(*old_modules) @unittest.skipIf(sys.platform != "win32", "Windows-only test") def test_sundry_windows(self): From webhook-mailer at python.org Tue Aug 4 12:30:21 2020 From: webhook-mailer at python.org (Mark Shannon) Date: Tue, 04 Aug 2020 16:30:21 -0000 Subject: [Python-checkins] bpo-41463: Generate information about jumps from 'opcode.py' rather than duplicating it in 'compile.c' (GH-21714) Message-ID: https://github.com/python/cpython/commit/582aaf19e8b94a70c1f96792197770d604ba0fdf commit: 582aaf19e8b94a70c1f96792197770d604ba0fdf branch: master author: Mark Shannon committer: GitHub date: 2020-08-04T17:30:11+01:00 summary: bpo-41463: Generate information about jumps from 'opcode.py' rather than duplicating it in 'compile.c' (GH-21714) Generate information about jumps from 'opcode.py' rather than duplicate it in 'compile.c' files: M Include/opcode.h M Python/compile.c M Tools/scripts/generate_opcode_h.py diff --git a/Include/opcode.h b/Include/opcode.h index 19944fac0b9f2..420c87aa0f24f 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -127,6 +127,28 @@ extern "C" { #define SET_UPDATE 163 #define DICT_MERGE 164 #define DICT_UPDATE 165 +#ifdef NEED_OPCODE_JUMP_TABLES +static uint32_t _PyOpcode_RelativeJump[8] = { + 0U, + 0U, + 536870912U, + 67125248U, + 67141632U, + 0U, + 0U, + 0U, +}; +static uint32_t _PyOpcode_Jump[8] = { + 0U, + 0U, + 536870912U, + 101695488U, + 67141632U, + 0U, + 0U, + 0U, +}; +#endif /* OPCODE_TABLES */ /* EXCEPT_HANDLER is a special, implicit block type which is created when entering an except handler. It is not an opcode but we define it here diff --git a/Python/compile.c b/Python/compile.c index 42b09fd96dfbb..5dbd9f221fdf1 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -27,6 +27,7 @@ #include "ast.h" #include "code.h" #include "symtable.h" +#define NEED_OPCODE_JUMP_TABLES #include "opcode.h" #include "wordcode_helpers.h" @@ -45,14 +46,38 @@ && (c->u->u_ste->ste_type == ModuleBlock)) struct instr { - unsigned i_jabs : 1; - unsigned i_jrel : 1; unsigned char i_opcode; int i_oparg; struct basicblock_ *i_target; /* target block (if jump instruction) */ int i_lineno; }; +#define LOG_BITS_PER_INT 5 +#define MASK_LOW_LOG_BITS 31 + +static inline int +is_bit_set_in_table(uint32_t *table, int bitindex) { + /* Is the relevant bit set in the relevant word? */ + /* 256 bits fit into 8 32-bits words. + * Word is indexed by (bitindex>>ln(size of int in bits)). + * Bit within word is the low bits of bitindex. + */ + uint32_t word = table[bitindex >> LOG_BITS_PER_INT]; + return (word >> (bitindex & MASK_LOW_LOG_BITS)) & 1; +} + +static inline int +is_relative_jump(struct instr *i) +{ + return is_bit_set_in_table(_PyOpcode_RelativeJump, i->i_opcode); +} + +static inline int +is_jump(struct instr *i) +{ + return is_bit_set_in_table(_PyOpcode_Jump, i->i_opcode); +} + typedef struct basicblock_ { /* Each basicblock in a compilation unit is linked via b_list in the reverse order that the block are allocated. b_list points to the next @@ -182,7 +207,7 @@ static basicblock *compiler_new_block(struct compiler *); static int compiler_next_instr(basicblock *); static int compiler_addop(struct compiler *, int); static int compiler_addop_i(struct compiler *, int, Py_ssize_t); -static int compiler_addop_j(struct compiler *, int, basicblock *, int); +static int compiler_addop_j(struct compiler *, int, basicblock *); static int compiler_error(struct compiler *, const char *); static int compiler_warn(struct compiler *, const char *, ...); static int compiler_nameop(struct compiler *, identifier, expr_context_ty); @@ -1381,7 +1406,7 @@ compiler_addop_i(struct compiler *c, int opcode, Py_ssize_t oparg) } static int -compiler_addop_j(struct compiler *c, int opcode, basicblock *b, int absolute) +compiler_addop_j(struct compiler *c, int opcode, basicblock *b) { struct instr *i; int off; @@ -1398,10 +1423,6 @@ compiler_addop_j(struct compiler *c, int opcode, basicblock *b, int absolute) i = &c->u->u_curblock->b_instr[off]; i->i_opcode = opcode; i->i_target = b; - if (absolute) - i->i_jabs = 1; - else - i->i_jrel = 1; i->i_lineno = c->u->u_lineno; return 1; } @@ -1471,17 +1492,11 @@ compiler_addop_j(struct compiler *c, int opcode, basicblock *b, int absolute) return 0; \ } -#define ADDOP_JABS(C, OP, O) { \ - if (!compiler_addop_j((C), (OP), (O), 1)) \ +#define ADDOP_JUMP(C, OP, O) { \ + if (!compiler_addop_j((C), (OP), (O))) \ return 0; \ } -#define ADDOP_JREL(C, OP, O) { \ - if (!compiler_addop_j((C), (OP), (O), 0)) \ - return 0; \ -} - - #define ADDOP_COMPARE(C, CMP) { \ if (!compiler_addcompare((C), (cmpop_ty)(CMP))) \ return 0; \ @@ -2545,7 +2560,7 @@ compiler_jump_if(struct compiler *c, expr_ty e, basicblock *next, int cond) return 0; if (!compiler_jump_if(c, e->v.IfExp.body, next, cond)) return 0; - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); compiler_use_next_block(c, next2); if (!compiler_jump_if(c, e->v.IfExp.orelse, next, cond)) return 0; @@ -2568,20 +2583,20 @@ compiler_jump_if(struct compiler *c, expr_ty e, basicblock *next, int cond) ADDOP(c, DUP_TOP); ADDOP(c, ROT_THREE); ADDOP_COMPARE(c, asdl_seq_GET(e->v.Compare.ops, i)); - ADDOP_JABS(c, POP_JUMP_IF_FALSE, cleanup); + ADDOP_JUMP(c, POP_JUMP_IF_FALSE, cleanup); NEXT_BLOCK(c); } VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n)); ADDOP_COMPARE(c, asdl_seq_GET(e->v.Compare.ops, n)); - ADDOP_JABS(c, cond ? POP_JUMP_IF_TRUE : POP_JUMP_IF_FALSE, next); + ADDOP_JUMP(c, cond ? POP_JUMP_IF_TRUE : POP_JUMP_IF_FALSE, next); basicblock *end = compiler_new_block(c); if (end == NULL) return 0; - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); compiler_use_next_block(c, cleanup); ADDOP(c, POP_TOP); if (!cond) { - ADDOP_JREL(c, JUMP_FORWARD, next); + ADDOP_JUMP(c, JUMP_FORWARD, next); } compiler_use_next_block(c, end); return 1; @@ -2596,7 +2611,7 @@ compiler_jump_if(struct compiler *c, expr_ty e, basicblock *next, int cond) /* general implementation */ VISIT(c, expr, e); - ADDOP_JABS(c, cond ? POP_JUMP_IF_TRUE : POP_JUMP_IF_FALSE, next); + ADDOP_JUMP(c, cond ? POP_JUMP_IF_TRUE : POP_JUMP_IF_FALSE, next); return 1; } @@ -2615,7 +2630,7 @@ compiler_ifexp(struct compiler *c, expr_ty e) if (!compiler_jump_if(c, e->v.IfExp.test, next, 0)) return 0; VISIT(c, expr, e->v.IfExp.body); - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); compiler_use_next_block(c, next); VISIT(c, expr, e->v.IfExp.orelse); compiler_use_next_block(c, end); @@ -2721,7 +2736,7 @@ compiler_if(struct compiler *c, stmt_ty s) } VISIT_SEQ(c, stmt, s->v.If.body); if (asdl_seq_LEN(s->v.If.orelse)) { - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); compiler_use_next_block(c, next); VISIT_SEQ(c, stmt, s->v.If.orelse); } @@ -2747,10 +2762,10 @@ compiler_for(struct compiler *c, stmt_ty s) VISIT(c, expr, s->v.For.iter); ADDOP(c, GET_ITER); compiler_use_next_block(c, start); - ADDOP_JREL(c, FOR_ITER, cleanup); + ADDOP_JUMP(c, FOR_ITER, cleanup); VISIT(c, expr, s->v.For.target); VISIT_SEQ(c, stmt, s->v.For.body); - ADDOP_JABS(c, JUMP_ABSOLUTE, start); + ADDOP_JUMP(c, JUMP_ABSOLUTE, start); compiler_use_next_block(c, cleanup); compiler_pop_fblock(c, FOR_LOOP, start); @@ -2786,7 +2801,7 @@ compiler_async_for(struct compiler *c, stmt_ty s) return 0; } /* SETUP_FINALLY to guard the __anext__ call */ - ADDOP_JREL(c, SETUP_FINALLY, except); + ADDOP_JUMP(c, SETUP_FINALLY, except); ADDOP(c, GET_ANEXT); ADDOP_LOAD_CONST(c, Py_None); ADDOP(c, YIELD_FROM); @@ -2795,7 +2810,7 @@ compiler_async_for(struct compiler *c, stmt_ty s) /* Success block for __anext__ */ VISIT(c, expr, s->v.AsyncFor.target); VISIT_SEQ(c, stmt, s->v.AsyncFor.body); - ADDOP_JABS(c, JUMP_ABSOLUTE, start); + ADDOP_JUMP(c, JUMP_ABSOLUTE, start); compiler_pop_fblock(c, FOR_LOOP, start); @@ -2859,7 +2874,7 @@ compiler_while(struct compiler *c, stmt_ty s) return 0; } VISIT_SEQ(c, stmt, s->v.While.body); - ADDOP_JABS(c, JUMP_ABSOLUTE, loop); + ADDOP_JUMP(c, JUMP_ABSOLUTE, loop); /* XXX should the two POP instructions be in a separate block if there is no else clause ? @@ -2918,7 +2933,7 @@ compiler_break(struct compiler *c) if (!compiler_unwind_fblock(c, loop, 0)) { return 0; } - ADDOP_JABS(c, JUMP_ABSOLUTE, loop->fb_exit); + ADDOP_JUMP(c, JUMP_ABSOLUTE, loop->fb_exit); return 1; } @@ -2932,7 +2947,7 @@ compiler_continue(struct compiler *c) if (loop == NULL) { return compiler_error(c, "'continue' not properly in loop"); } - ADDOP_JABS(c, JUMP_ABSOLUTE, loop->fb_block); + ADDOP_JUMP(c, JUMP_ABSOLUTE, loop->fb_block); return 1; } @@ -2978,7 +2993,7 @@ compiler_try_finally(struct compiler *c, stmt_ty s) return 0; /* `try` block */ - ADDOP_JREL(c, SETUP_FINALLY, end); + ADDOP_JUMP(c, SETUP_FINALLY, end); compiler_use_next_block(c, body); if (!compiler_push_fblock(c, FINALLY_TRY, body, end, s->v.Try.finalbody)) return 0; @@ -2992,7 +3007,7 @@ compiler_try_finally(struct compiler *c, stmt_ty s) ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, FINALLY_TRY, body); VISIT_SEQ(c, stmt, s->v.Try.finalbody); - ADDOP_JREL(c, JUMP_FORWARD, exit); + ADDOP_JUMP(c, JUMP_FORWARD, exit); /* `finally` block */ compiler_use_next_block(c, end); if (!compiler_push_fblock(c, FINALLY_END, end, NULL, NULL)) @@ -3046,14 +3061,14 @@ compiler_try_except(struct compiler *c, stmt_ty s) end = compiler_new_block(c); if (body == NULL || except == NULL || orelse == NULL || end == NULL) return 0; - ADDOP_JREL(c, SETUP_FINALLY, except); + ADDOP_JUMP(c, SETUP_FINALLY, except); compiler_use_next_block(c, body); if (!compiler_push_fblock(c, EXCEPT, body, NULL, NULL)) return 0; VISIT_SEQ(c, stmt, s->v.Try.body); ADDOP(c, POP_BLOCK); compiler_pop_fblock(c, EXCEPT, body); - ADDOP_JREL(c, JUMP_FORWARD, orelse); + ADDOP_JUMP(c, JUMP_FORWARD, orelse); n = asdl_seq_LEN(s->v.Try.handlers); compiler_use_next_block(c, except); for (i = 0; i < n; i++) { @@ -3068,7 +3083,7 @@ compiler_try_except(struct compiler *c, stmt_ty s) if (handler->v.ExceptHandler.type) { ADDOP(c, DUP_TOP); VISIT(c, expr, handler->v.ExceptHandler.type); - ADDOP_JABS(c, JUMP_IF_NOT_EXC_MATCH, except); + ADDOP_JUMP(c, JUMP_IF_NOT_EXC_MATCH, except); } ADDOP(c, POP_TOP); if (handler->v.ExceptHandler.name) { @@ -3095,7 +3110,7 @@ compiler_try_except(struct compiler *c, stmt_ty s) */ /* second try: */ - ADDOP_JREL(c, SETUP_FINALLY, cleanup_end); + ADDOP_JUMP(c, SETUP_FINALLY, cleanup_end); compiler_use_next_block(c, cleanup_body); if (!compiler_push_fblock(c, HANDLER_CLEANUP, cleanup_body, NULL, handler->v.ExceptHandler.name)) return 0; @@ -3109,7 +3124,7 @@ compiler_try_except(struct compiler *c, stmt_ty s) ADDOP_LOAD_CONST(c, Py_None); compiler_nameop(c, handler->v.ExceptHandler.name, Store); compiler_nameop(c, handler->v.ExceptHandler.name, Del); - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); /* except: */ compiler_use_next_block(c, cleanup_end); @@ -3136,7 +3151,7 @@ compiler_try_except(struct compiler *c, stmt_ty s) VISIT_SEQ(c, stmt, handler->v.ExceptHandler.body); compiler_pop_fblock(c, HANDLER_CLEANUP, cleanup_body); ADDOP(c, POP_EXCEPT); - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); } compiler_use_next_block(c, except); } @@ -3645,7 +3660,7 @@ compiler_boolop(struct compiler *c, expr_ty e) assert(n >= 0); for (i = 0; i < n; ++i) { VISIT(c, expr, (expr_ty)asdl_seq_GET(s, i)); - ADDOP_JABS(c, jumpi, end); + ADDOP_JUMP(c, jumpi, end); basicblock *next = compiler_new_block(c); if (next == NULL) { return 0; @@ -3933,7 +3948,7 @@ compiler_compare(struct compiler *c, expr_ty e) ADDOP(c, DUP_TOP); ADDOP(c, ROT_THREE); ADDOP_COMPARE(c, asdl_seq_GET(e->v.Compare.ops, i)); - ADDOP_JABS(c, JUMP_IF_FALSE_OR_POP, cleanup); + ADDOP_JUMP(c, JUMP_IF_FALSE_OR_POP, cleanup); NEXT_BLOCK(c); } VISIT(c, expr, (expr_ty)asdl_seq_GET(e->v.Compare.comparators, n)); @@ -3941,7 +3956,7 @@ compiler_compare(struct compiler *c, expr_ty e) basicblock *end = compiler_new_block(c); if (end == NULL) return 0; - ADDOP_JREL(c, JUMP_FORWARD, end); + ADDOP_JUMP(c, JUMP_FORWARD, end); compiler_use_next_block(c, cleanup); ADDOP(c, ROT_TWO); ADDOP(c, POP_TOP); @@ -4435,7 +4450,7 @@ compiler_sync_comprehension_generator(struct compiler *c, if (start) { depth++; compiler_use_next_block(c, start); - ADDOP_JREL(c, FOR_ITER, anchor); + ADDOP_JUMP(c, FOR_ITER, anchor); NEXT_BLOCK(c); } VISIT(c, expr, gen->target); @@ -4487,7 +4502,7 @@ compiler_sync_comprehension_generator(struct compiler *c, } compiler_use_next_block(c, if_cleanup); if (start) { - ADDOP_JABS(c, JUMP_ABSOLUTE, start); + ADDOP_JUMP(c, JUMP_ABSOLUTE, start); compiler_use_next_block(c, anchor); } @@ -4526,7 +4541,7 @@ compiler_async_comprehension_generator(struct compiler *c, compiler_use_next_block(c, start); - ADDOP_JREL(c, SETUP_FINALLY, except); + ADDOP_JUMP(c, SETUP_FINALLY, except); ADDOP(c, GET_ANEXT); ADDOP_LOAD_CONST(c, Py_None); ADDOP(c, YIELD_FROM); @@ -4577,7 +4592,7 @@ compiler_async_comprehension_generator(struct compiler *c, } } compiler_use_next_block(c, if_cleanup); - ADDOP_JABS(c, JUMP_ABSOLUTE, start); + ADDOP_JUMP(c, JUMP_ABSOLUTE, start); compiler_use_next_block(c, except); ADDOP(c, END_ASYNC_FOR); @@ -4773,7 +4788,7 @@ compiler_with_except_finish(struct compiler *c) { exit = compiler_new_block(c); if (exit == NULL) return 0; - ADDOP_JABS(c, POP_JUMP_IF_TRUE, exit); + ADDOP_JUMP(c, POP_JUMP_IF_TRUE, exit); ADDOP(c, RERAISE); compiler_use_next_block(c, exit); ADDOP(c, POP_TOP); @@ -4835,7 +4850,7 @@ compiler_async_with(struct compiler *c, stmt_ty s, int pos) ADDOP_LOAD_CONST(c, Py_None); ADDOP(c, YIELD_FROM); - ADDOP_JREL(c, SETUP_ASYNC_WITH, final); + ADDOP_JUMP(c, SETUP_ASYNC_WITH, final); /* SETUP_ASYNC_WITH pushes a finally block. */ compiler_use_next_block(c, block); @@ -4873,7 +4888,7 @@ compiler_async_with(struct compiler *c, stmt_ty s, int pos) ADDOP(c, POP_TOP); - ADDOP_JABS(c, JUMP_ABSOLUTE, exit); + ADDOP_JUMP(c, JUMP_ABSOLUTE, exit); /* For exceptional outcome: */ compiler_use_next_block(c, final); @@ -4927,7 +4942,7 @@ compiler_with(struct compiler *c, stmt_ty s, int pos) /* Evaluate EXPR */ VISIT(c, expr, item->context_expr); /* Will push bound __exit__ */ - ADDOP_JREL(c, SETUP_WITH, final); + ADDOP_JUMP(c, SETUP_WITH, final); /* SETUP_WITH pushes a finally block. */ compiler_use_next_block(c, block); @@ -4961,7 +4976,7 @@ compiler_with(struct compiler *c, stmt_ty s, int pos) if (!compiler_call_exit_with_nones(c)) return 0; ADDOP(c, POP_TOP); - ADDOP_JREL(c, JUMP_FORWARD, exit); + ADDOP_JUMP(c, JUMP_FORWARD, exit); /* For exceptional outcome: */ compiler_use_next_block(c, final); @@ -5492,7 +5507,7 @@ stackdepth(struct compiler *c) maxdepth = new_depth; } assert(depth >= 0); /* invalid code or bug in stackdepth() */ - if (instr->i_jrel || instr->i_jabs) { + if (is_jump(instr)) { effect = stack_effect(instr->i_opcode, instr->i_oparg, 1); assert(effect != PY_INVALID_STACK_EFFECT); int target_depth = depth + effect; @@ -5730,9 +5745,9 @@ assemble_jump_offsets(struct assembler *a, struct compiler *c) the jump instruction. */ bsize += isize; - if (instr->i_jabs || instr->i_jrel) { + if (is_jump(instr)) { instr->i_oparg = instr->i_target->b_offset; - if (instr->i_jrel) { + if (is_relative_jump(instr)) { instr->i_oparg -= bsize; } instr->i_oparg *= sizeof(_Py_CODEUNIT); @@ -5946,8 +5961,8 @@ makecode(struct compiler *c, struct assembler *a, PyObject *consts) static void dump_instr(const struct instr *i) { - const char *jrel = i->i_jrel ? "jrel " : ""; - const char *jabs = i->i_jabs ? "jabs " : ""; + const char *jrel = (is_relative_jump(instr)) ? "jrel " : ""; + const char *jabs = (is_jump(instr) && !is_relative_jump(instr))? "jabs " : ""; char arg[128]; *arg = '\0'; @@ -6122,7 +6137,7 @@ optimize_basic_block(basicblock *bb, PyObject *consts) struct instr *inst = &bb->b_instr[i]; int oparg = inst->i_oparg; int nextop = i+1 < bb->b_iused ? bb->b_instr[i+1].i_opcode : 0; - if (inst->i_jabs || inst->i_jrel) { + if (is_jump(inst)) { /* Skip over empty basic blocks. */ while (inst->i_target->b_iused == 0) { inst->i_target = inst->i_target->b_next; @@ -6148,7 +6163,6 @@ optimize_basic_block(basicblock *bb, PyObject *consts) if (is_true == 1) { inst->i_opcode = NOP; bb->b_instr[i+1].i_opcode = NOP; - bb->b_instr[i+1].i_jabs = 0; } break; @@ -6318,7 +6332,7 @@ mark_reachable(struct assembler *a) { } for (int i = 0; i < b->b_iused; i++) { basicblock *target; - if (b->b_instr[i].i_jrel || b->b_instr[i].i_jabs) { + if (is_jump(&b->b_instr[i])) { target = b->b_instr[i].i_target; if (target->b_reachable == 0) { target->b_reachable = 1; diff --git a/Tools/scripts/generate_opcode_h.py b/Tools/scripts/generate_opcode_h.py index 873f82156e217..cba13b2421390 100644 --- a/Tools/scripts/generate_opcode_h.py +++ b/Tools/scripts/generate_opcode_h.py @@ -30,6 +30,18 @@ #endif /* !Py_OPCODE_H */ """ +UINT32_MASK = (1<<32)-1 + +def write_int_array_from_ops(name, ops, out): + bits = 0 + for op in ops: + bits |= 1<>= 32 + assert bits == 0 + out.write(f"}};\n") def main(opcode_py, outfile='Include/opcode.h'): opcode = {} @@ -41,6 +53,8 @@ def main(opcode_py, outfile='Include/opcode.h'): code = fp.read() exec(code, opcode) opmap = opcode['opmap'] + hasjrel = opcode['hasjrel'] + hasjabs = opcode['hasjabs'] with open(outfile, 'w') as fobj: fobj.write(header) for name in opcode['opname']: @@ -49,8 +63,13 @@ def main(opcode_py, outfile='Include/opcode.h'): if name == 'POP_EXCEPT': # Special entry for HAVE_ARGUMENT fobj.write("#define %-23s %3d\n" % ('HAVE_ARGUMENT', opcode['HAVE_ARGUMENT'])) + fobj.write("#ifdef NEED_OPCODE_JUMP_TABLES\n") + write_int_array_from_ops("_PyOpcode_RelativeJump", opcode['hasjrel'], fobj) + write_int_array_from_ops("_PyOpcode_Jump", opcode['hasjrel'] + opcode['hasjabs'], fobj) + fobj.write("#endif /* OPCODE_TABLES */\n") fobj.write(footer) + print("%s regenerated from %s" % (outfile, opcode_py)) From webhook-mailer at python.org Tue Aug 4 21:49:19 2020 From: webhook-mailer at python.org (Inada Naoki) Date: Wed, 05 Aug 2020 01:49:19 -0000 Subject: [Python-checkins] bpo-36346: Doc: Update removal schedule of legacy Unicode (GH-21479) Message-ID: https://github.com/python/cpython/commit/270b4ad4df795783d417ba15080da8f95e598689 commit: 270b4ad4df795783d417ba15080da8f95e598689 branch: master author: Inada Naoki committer: GitHub date: 2020-08-05T10:48:51+09:00 summary: bpo-36346: Doc: Update removal schedule of legacy Unicode (GH-21479) See PEP 623 for detail. files: M Doc/c-api/unicode.rst diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 0748a1e319489..f3f0c4c6c2b9b 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -34,6 +34,11 @@ can internally be in two states depending on how they were created: :c:type:`Py_UNICODE*` representation; you will have to call :c:func:`PyUnicode_READY` on them before calling any other API. +.. note:: + The "legacy" Unicode object will be removed in Python 3.12 with deprecated + APIs. All Unicode objects will be "canonical" since then. See :pep:`623` + for more information. + Unicode Type """""""""""" @@ -107,6 +112,9 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 + .. deprecated-removed:: 3.10 3.12 + This API will be removed with :c:func:`PyUnicode_FromUnicode`. + .. c:function:: Py_ssize_t PyUnicode_GET_LENGTH(PyObject *o) @@ -138,6 +146,9 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 + .. deprecated-removed:: 3.10 3.12 + ``PyUnicode_WCHAR_KIND`` is deprecated. + .. c:function:: int PyUnicode_KIND(PyObject *o) @@ -203,7 +214,7 @@ access internal read-only data of Unicode objects: code units (this includes surrogate pairs as 2 units). *o* has to be a Unicode object (not checked). - .. deprecated-removed:: 3.3 4.0 + .. deprecated-removed:: 3.3 3.12 Part of the old-style Unicode API, please migrate to using :c:func:`PyUnicode_GET_LENGTH`. @@ -213,7 +224,7 @@ access internal read-only data of Unicode objects: Return the size of the deprecated :c:type:`Py_UNICODE` representation in bytes. *o* has to be a Unicode object (not checked). - .. deprecated-removed:: 3.3 4.0 + .. deprecated-removed:: 3.3 3.12 Part of the old-style Unicode API, please migrate to using :c:func:`PyUnicode_GET_LENGTH`. @@ -235,7 +246,7 @@ access internal read-only data of Unicode objects: code to use the new :c:func:`PyUnicode_nBYTE_DATA` macros or use :c:func:`PyUnicode_WRITE` or :c:func:`PyUnicode_READ`. - .. deprecated-removed:: 3.3 4.0 + .. deprecated-removed:: 3.3 3.12 Part of the old-style Unicode API, please migrate to using the :c:func:`PyUnicode_nBYTE_DATA` family of macros. @@ -687,8 +698,10 @@ Extension modules can continue using them, as they will not be removed in Python string content has been filled before using any of the access macros such as :c:func:`PyUnicode_KIND`. - Please migrate to using :c:func:`PyUnicode_FromKindAndData`, - :c:func:`PyUnicode_FromWideChar` or :c:func:`PyUnicode_New`. + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_FromKindAndData`, :c:func:`PyUnicode_FromWideChar`, or + :c:func:`PyUnicode_New`. .. c:function:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode) @@ -701,9 +714,10 @@ Extension modules can continue using them, as they will not be removed in Python embedded null code points, which would cause the string to be truncated when used in most C functions. - Please migrate to using :c:func:`PyUnicode_AsUCS4`, - :c:func:`PyUnicode_AsWideChar`, :c:func:`PyUnicode_ReadChar` or similar new - APIs. + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsWideChar`, + :c:func:`PyUnicode_ReadChar` or similar new APIs. .. c:function:: PyObject* PyUnicode_TransformDecimalToASCII(Py_UNICODE *s, Py_ssize_t size) @@ -723,13 +737,20 @@ Extension modules can continue using them, as they will not be removed in Python .. versionadded:: 3.3 + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsWideChar`, + :c:func:`PyUnicode_ReadChar` or similar new APIs. + .. c:function:: Py_ssize_t PyUnicode_GetSize(PyObject *unicode) Return the size of the deprecated :c:type:`Py_UNICODE` representation, in code units (this includes surrogate pairs as 2 units). - Please migrate to using :c:func:`PyUnicode_GetLength`. + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_GET_LENGTH`. .. c:function:: PyObject* PyUnicode_FromObject(PyObject *obj) From webhook-mailer at python.org Tue Aug 4 21:57:14 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 05 Aug 2020 01:57:14 -0000 Subject: [Python-checkins] bpo-36346: Doc: Update removal schedule of legacy Unicode (GH-21479) Message-ID: https://github.com/python/cpython/commit/f0e030cacb940f061e0b09efbffc2fd984b95259 commit: f0e030cacb940f061e0b09efbffc2fd984b95259 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-04T18:57:10-07:00 summary: bpo-36346: Doc: Update removal schedule of legacy Unicode (GH-21479) See PEP 623 for detail. (cherry picked from commit 270b4ad4df795783d417ba15080da8f95e598689) Co-authored-by: Inada Naoki files: M Doc/c-api/unicode.rst diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index b261efe2ba477..a70b1d0afd2d5 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -34,6 +34,11 @@ can internally be in two states depending on how they were created: :c:type:`Py_UNICODE*` representation; you will have to call :c:func:`PyUnicode_READY` on them before calling any other API. +.. note:: + The "legacy" Unicode object will be removed in Python 3.12 with deprecated + APIs. All Unicode objects will be "canonical" since then. See :pep:`623` + for more information. + Unicode Type """""""""""" @@ -107,6 +112,9 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 + .. deprecated-removed:: 3.10 3.12 + This API will be removed with :c:func:`PyUnicode_FromUnicode`. + .. c:function:: Py_ssize_t PyUnicode_GET_LENGTH(PyObject *o) @@ -138,6 +146,9 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 + .. deprecated-removed:: 3.10 3.12 + ``PyUnicode_WCHAR_KIND`` is deprecated. + .. c:function:: int PyUnicode_KIND(PyObject *o) @@ -208,7 +219,7 @@ access internal read-only data of Unicode objects: code units (this includes surrogate pairs as 2 units). *o* has to be a Unicode object (not checked). - .. deprecated-removed:: 3.3 4.0 + .. deprecated-removed:: 3.3 3.12 Part of the old-style Unicode API, please migrate to using :c:func:`PyUnicode_GET_LENGTH`. @@ -218,7 +229,7 @@ access internal read-only data of Unicode objects: Return the size of the deprecated :c:type:`Py_UNICODE` representation in bytes. *o* has to be a Unicode object (not checked). - .. deprecated-removed:: 3.3 4.0 + .. deprecated-removed:: 3.3 3.12 Part of the old-style Unicode API, please migrate to using :c:func:`PyUnicode_GET_LENGTH`. @@ -240,7 +251,7 @@ access internal read-only data of Unicode objects: code to use the new :c:func:`PyUnicode_nBYTE_DATA` macros or use :c:func:`PyUnicode_WRITE` or :c:func:`PyUnicode_READ`. - .. deprecated-removed:: 3.3 4.0 + .. deprecated-removed:: 3.3 3.12 Part of the old-style Unicode API, please migrate to using the :c:func:`PyUnicode_nBYTE_DATA` family of macros. @@ -682,8 +693,10 @@ Extension modules can continue using them, as they will not be removed in Python string content has been filled before using any of the access macros such as :c:func:`PyUnicode_KIND`. - Please migrate to using :c:func:`PyUnicode_FromKindAndData`, - :c:func:`PyUnicode_FromWideChar` or :c:func:`PyUnicode_New`. + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_FromKindAndData`, :c:func:`PyUnicode_FromWideChar`, or + :c:func:`PyUnicode_New`. .. c:function:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode) @@ -696,9 +709,10 @@ Extension modules can continue using them, as they will not be removed in Python embedded null code points, which would cause the string to be truncated when used in most C functions. - Please migrate to using :c:func:`PyUnicode_AsUCS4`, - :c:func:`PyUnicode_AsWideChar`, :c:func:`PyUnicode_ReadChar` or similar new - APIs. + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsWideChar`, + :c:func:`PyUnicode_ReadChar` or similar new APIs. .. deprecated-removed:: 3.3 3.10 @@ -720,6 +734,11 @@ Extension modules can continue using them, as they will not be removed in Python .. versionadded:: 3.3 + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsWideChar`, + :c:func:`PyUnicode_ReadChar` or similar new APIs. + .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode) @@ -740,7 +759,9 @@ Extension modules can continue using them, as they will not be removed in Python Return the size of the deprecated :c:type:`Py_UNICODE` representation, in code units (this includes surrogate pairs as 2 units). - Please migrate to using :c:func:`PyUnicode_GetLength`. + .. deprecated-removed:: 3.3 3.12 + Part of the old-style Unicode API, please migrate to using + :c:func:`PyUnicode_GET_LENGTH`. .. c:function:: PyObject* PyUnicode_FromObject(PyObject *obj) From webhook-mailer at python.org Wed Aug 5 09:33:01 2020 From: webhook-mailer at python.org (Batuhan Taskaya) Date: Wed, 05 Aug 2020 13:33:01 -0000 Subject: [Python-checkins] bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-20312) Message-ID: https://github.com/python/cpython/commit/8f4380d2f5839a321475104765221a7394a9d649 commit: 8f4380d2f5839a321475104765221a7394a9d649 branch: master author: Batuhan Taskaya committer: GitHub date: 2020-08-05T14:32:32+01:00 summary: bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-20312) files: A Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst M Lib/ast.py M Lib/test/test_ast.py diff --git a/Lib/ast.py b/Lib/ast.py index 6a5b39e270b9b..65ebd0100de00 100644 --- a/Lib/ast.py +++ b/Lib/ast.py @@ -180,7 +180,11 @@ def copy_location(new_node, old_node): for attr in 'lineno', 'col_offset', 'end_lineno', 'end_col_offset': if attr in old_node._attributes and attr in new_node._attributes: value = getattr(old_node, attr, None) - if value is not None: + # end_lineno and end_col_offset are optional attributes, and they + # should be copied whether the value is None or not. + if value is not None or ( + hasattr(old_node, attr) and attr.startswith("end_") + ): setattr(new_node, attr, value) return new_node @@ -229,8 +233,11 @@ def increment_lineno(node, n=1): for child in walk(node): if 'lineno' in child._attributes: child.lineno = getattr(child, 'lineno', 0) + n - if 'end_lineno' in child._attributes: - child.end_lineno = getattr(child, 'end_lineno', 0) + n + if ( + "end_lineno" in child._attributes + and (end_lineno := getattr(child, "end_lineno", 0)) is not None + ): + child.end_lineno = end_lineno + n return node diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 78e4a5653d4ef..f5aef61ec6f7c 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -812,6 +812,12 @@ def test_copy_location(self): 'lineno=1, col_offset=4, end_lineno=1, end_col_offset=5), lineno=1, ' 'col_offset=0, end_lineno=1, end_col_offset=5))' ) + src = ast.Call(col_offset=1, lineno=1, end_lineno=1, end_col_offset=1) + new = ast.copy_location(src, ast.Call(col_offset=None, lineno=None)) + self.assertIsNone(new.end_lineno) + self.assertIsNone(new.end_col_offset) + self.assertEqual(new.lineno, 1) + self.assertEqual(new.col_offset, 1) def test_fix_missing_locations(self): src = ast.parse('write("spam")') @@ -851,6 +857,11 @@ def test_increment_lineno(self): 'lineno=4, col_offset=4, end_lineno=4, end_col_offset=5), lineno=4, ' 'col_offset=0, end_lineno=4, end_col_offset=5))' ) + src = ast.Call( + func=ast.Name("test", ast.Load()), args=[], keywords=[], lineno=1 + ) + self.assertEqual(ast.increment_lineno(src).lineno, 2) + self.assertIsNone(ast.increment_lineno(src).end_lineno) def test_iter_fields(self): node = ast.parse('foo()', mode='eval') diff --git a/Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst b/Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst new file mode 100644 index 0000000000000..7409eb3d80df6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst @@ -0,0 +1,2 @@ +Handle cases where the ``end_lineno`` is ``None`` on +:func:`ast.increment_lineno`. From webhook-mailer at python.org Wed Aug 5 10:23:18 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 05 Aug 2020 14:23:18 -0000 Subject: [Python-checkins] bpo-40989: Fix compiler warning in winreg.c (GH-21722) Message-ID: https://github.com/python/cpython/commit/15edaecd97a3f8e82895046462342d8ddd8b9f1b commit: 15edaecd97a3f8e82895046462342d8ddd8b9f1b branch: master author: Victor Stinner committer: GitHub date: 2020-08-05T16:23:10+02:00 summary: bpo-40989: Fix compiler warning in winreg.c (GH-21722) Explicitly cast PyHKEYObject* to PyObject* to call _PyObject_Init(). files: M PC/winreg.c diff --git a/PC/winreg.c b/PC/winreg.c index a24d784c773c0..78c08693a8ace 100644 --- a/PC/winreg.c +++ b/PC/winreg.c @@ -463,7 +463,7 @@ PyHKEY_FromHKEY(HKEY h) if (op == NULL) { return PyErr_NoMemory(); } - _PyObject_Init(op, &PyHKEY_Type); + _PyObject_Init((PyObject*)op, &PyHKEY_Type); op->hkey = h; return (PyObject *)op; } From webhook-mailer at python.org Wed Aug 5 14:37:28 2020 From: webhook-mailer at python.org (Batuhan Taskaya) Date: Wed, 05 Aug 2020 18:37:28 -0000 Subject: [Python-checkins] [3.8] bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-21745) Message-ID: https://github.com/python/cpython/commit/b24c9d2b0656764bef48120d9511faf833bd7ead commit: b24c9d2b0656764bef48120d9511faf833bd7ead branch: 3.8 author: Batuhan Taskaya committer: GitHub date: 2020-08-05T11:37:19-07:00 summary: [3.8] bpo-40726: handle uninitalized end_lineno on ast.increment_lineno (GH-21745) ?no (GH-20312). (cherry picked from commit 8f4380d2f5839a321475104765221a7394a9d649) Co-authored-by: Batuhan Taskaya Automerge-Triggered-By: @pablogsal files: A Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst M Lib/ast.py M Lib/test/test_ast.py diff --git a/Lib/ast.py b/Lib/ast.py index 99a1148a3ba24..d29db80a085fd 100644 --- a/Lib/ast.py +++ b/Lib/ast.py @@ -144,9 +144,14 @@ def copy_location(new_node, old_node): attributes) from *old_node* to *new_node* if possible, and return *new_node*. """ for attr in 'lineno', 'col_offset', 'end_lineno', 'end_col_offset': - if attr in old_node._attributes and attr in new_node._attributes \ - and hasattr(old_node, attr): - setattr(new_node, attr, getattr(old_node, attr)) + if attr in old_node._attributes and attr in new_node._attributes: + value = getattr(old_node, attr, None) + # end_lineno and end_col_offset are optional attributes, and they + # should be copied whether the value is None or not. + if value is not None or ( + hasattr(old_node, attr) and attr.startswith("end_") + ): + setattr(new_node, attr, value) return new_node @@ -194,8 +199,11 @@ def increment_lineno(node, n=1): for child in walk(node): if 'lineno' in child._attributes: child.lineno = getattr(child, 'lineno', 0) + n - if 'end_lineno' in child._attributes: - child.end_lineno = getattr(child, 'end_lineno', 0) + n + if ( + "end_lineno" in child._attributes + and (end_lineno := getattr(child, "end_lineno", 0)) is not None + ): + child.end_lineno = end_lineno + n return node diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index b921f4a5d6826..c625e693f50de 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -718,6 +718,15 @@ def test_copy_location(self): 'lineno=1, col_offset=4, end_lineno=1, end_col_offset=5), lineno=1, ' 'col_offset=0, end_lineno=1, end_col_offset=5))' ) + src = ast.Call(col_offset=1, lineno=1, end_lineno=1, end_col_offset=1) + new = ast.copy_location(src, ast.Call( + col_offset=None, lineno=None, + end_lineno=None, end_col_offset=None + )) + self.assertIsNone(new.end_lineno) + self.assertIsNone(new.end_col_offset) + self.assertEqual(new.lineno, 1) + self.assertEqual(new.col_offset, 1) def test_fix_missing_locations(self): src = ast.parse('write("spam")') @@ -757,6 +766,12 @@ def test_increment_lineno(self): 'lineno=4, col_offset=4, end_lineno=4, end_col_offset=5), lineno=4, ' 'col_offset=0, end_lineno=4, end_col_offset=5))' ) + src = ast.Call( + func=ast.Name("test", ast.Load()), args=[], keywords=[], + lineno=1, end_lineno=None + ) + self.assertEqual(ast.increment_lineno(src).lineno, 2) + self.assertIsNone(ast.increment_lineno(src).end_lineno) def test_iter_fields(self): node = ast.parse('foo()', mode='eval') diff --git a/Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst b/Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst new file mode 100644 index 0000000000000..7409eb3d80df6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-05-22-12-45-58.bpo-40726.7oBdMw.rst @@ -0,0 +1,2 @@ +Handle cases where the ``end_lineno`` is ``None`` on +:func:`ast.increment_lineno`. From webhook-mailer at python.org Wed Aug 5 14:45:01 2020 From: webhook-mailer at python.org (Eric L. Frederich) Date: Wed, 05 Aug 2020 18:45:01 -0000 Subject: [Python-checkins] bpo-41482: Fix error in ipaddress.IPv4Network docstring (GH-21736) Message-ID: https://github.com/python/cpython/commit/52f98424a55e14f05dfa7483cc0faf634a61c9ff commit: 52f98424a55e14f05dfa7483cc0faf634a61c9ff branch: master author: Eric L. Frederich committer: GitHub date: 2020-08-05T14:44:53-04:00 summary: bpo-41482: Fix error in ipaddress.IPv4Network docstring (GH-21736) files: M Lib/ipaddress.py diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index 75b4c352c1d25..bc662c415b2a4 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1466,7 +1466,7 @@ def __init__(self, address, strict=True): address: A string or integer representing the IP [& network]. '192.0.2.0/24' '192.0.2.0/255.255.255.0' - '192.0.0.2/0.0.0.255' + '192.0.2.0/0.0.0.255' are all functionally the same in IPv4. Similarly, '192.0.2.1' '192.0.2.1/255.255.255.255' From webhook-mailer at python.org Wed Aug 5 17:43:21 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 05 Aug 2020 21:43:21 -0000 Subject: [Python-checkins] bpo-41482: Fix error in ipaddress.IPv4Network docstring (GH-21736) Message-ID: https://github.com/python/cpython/commit/b5789a7419655f66692fab463320048bd2290e81 commit: b5789a7419655f66692fab463320048bd2290e81 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-05T14:43:16-07:00 summary: bpo-41482: Fix error in ipaddress.IPv4Network docstring (GH-21736) (cherry picked from commit 52f98424a55e14f05dfa7483cc0faf634a61c9ff) Co-authored-by: Eric L. Frederich files: M Lib/ipaddress.py diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index a3a04f7f4b309..1014eba551719 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1416,7 +1416,7 @@ def __init__(self, address, strict=True): address: A string or integer representing the IP [& network]. '192.0.2.0/24' '192.0.2.0/255.255.255.0' - '192.0.0.2/0.0.0.255' + '192.0.2.0/0.0.0.255' are all functionally the same in IPv4. Similarly, '192.0.2.1' '192.0.2.1/255.255.255.255' From webhook-mailer at python.org Thu Aug 6 07:51:34 2020 From: webhook-mailer at python.org (Hai Shi) Date: Thu, 06 Aug 2020 11:51:34 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21743) Message-ID: https://github.com/python/cpython/commit/79bb2c93f2d81702fdf1f93720369e18a76b7d1a commit: 79bb2c93f2d81702fdf1f93720369e18a76b7d1a branch: master author: Hai Shi committer: GitHub date: 2020-08-06T13:51:29+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21743) files: M Lib/test/test_dbm_gnu.py M Lib/test/test_faulthandler.py M Lib/test/test_grp.py M Lib/test/test_http_cookiejar.py M Lib/test/test_httpservers.py M Lib/test/test_import/__init__.py M Lib/test/test_ntpath.py M Lib/test/test_site.py M Lib/test/test_socketserver.py M Lib/test/test_statistics.py M Lib/test/test_tabnanny.py M Lib/test/test_tcl.py M Lib/test/test_tools/test_md5sum.py M Lib/test/test_turtle.py M Lib/test/test_urllib2net.py M Lib/test/test_urllibnet.py M Lib/test/test_webbrowser.py M Lib/test/test_winsound.py M Lib/tkinter/test/test_tkinter/test_images.py M Lib/tkinter/test/test_tkinter/test_loadtk.py diff --git a/Lib/test/test_dbm_gnu.py b/Lib/test/test_dbm_gnu.py index 078bf0e9b9270..017d0ffa658bd 100644 --- a/Lib/test/test_dbm_gnu.py +++ b/Lib/test/test_dbm_gnu.py @@ -3,7 +3,7 @@ gdbm = import_helper.import_module("dbm.gnu") #skip if not supported import unittest import os -from test.support import TESTFN, TESTFN_NONASCII, unlink +from test.support.os_helper import TESTFN, TESTFN_NONASCII, unlink filename = TESTFN diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py index c64afe88c25d6..80c1f7d90adf0 100644 --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -7,6 +7,7 @@ import sys import sysconfig from test import support +from test.support import os_helper from test.support import script_helper, is_android import tempfile import unittest @@ -51,7 +52,7 @@ def temporary_filename(): try: yield filename finally: - support.unlink(filename) + os_helper.unlink(filename) class FaultHandlerTests(unittest.TestCase): def get_output(self, code, filename=None, fd=None): diff --git a/Lib/test/test_grp.py b/Lib/test/test_grp.py index 0993f091f5956..c7ec03ec0e438 100644 --- a/Lib/test/test_grp.py +++ b/Lib/test/test_grp.py @@ -1,9 +1,10 @@ """Test script for the grp module.""" import unittest -from test import support +from test.support import import_helper -grp = support.import_module('grp') + +grp = import_helper.import_module('grp') class GroupDatabaseTestCase(unittest.TestCase): diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 2d7077af6da39..99d038fa15c1e 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -3,6 +3,8 @@ import os import re import test.support +from test.support import os_helper +from test.support import warnings_helper import time import unittest import urllib.request @@ -328,12 +330,12 @@ def _interact(cookiejar, url, set_cookie_hdrs, hdr_name): class FileCookieJarTests(unittest.TestCase): def test_constructor_with_str(self): - filename = test.support.TESTFN + filename = os_helper.TESTFN c = LWPCookieJar(filename) self.assertEqual(c.filename, filename) def test_constructor_with_path_like(self): - filename = pathlib.Path(test.support.TESTFN) + filename = pathlib.Path(os_helper.TESTFN) c = LWPCookieJar(filename) self.assertEqual(c.filename, os.fspath(filename)) @@ -353,7 +355,7 @@ class A: def test_lwp_valueless_cookie(self): # cookies with no value should be saved and loaded consistently - filename = test.support.TESTFN + filename = os_helper.TESTFN c = LWPCookieJar() interact_netscape(c, "http://www.acme.com/", 'boo') self.assertEqual(c._cookies["www.acme.com"]["/"]["boo"].value, None) @@ -368,7 +370,7 @@ def test_lwp_valueless_cookie(self): def test_bad_magic(self): # OSErrors (eg. file doesn't exist) are allowed to propagate - filename = test.support.TESTFN + filename = os_helper.TESTFN for cookiejar_class in LWPCookieJar, MozillaCookieJar: c = cookiejar_class() try: @@ -475,7 +477,7 @@ def test_domain_return_ok(self): def test_missing_value(self): # missing = sign in Cookie: header is regarded by Mozilla as a missing # name, and by http.cookiejar as a missing value - filename = test.support.TESTFN + filename = os_helper.TESTFN c = MozillaCookieJar(filename) interact_netscape(c, "http://www.acme.com/", 'eggs') interact_netscape(c, "http://www.acme.com/", '"spam"; path=/foo/') @@ -599,7 +601,7 @@ def test_expires(self): c = CookieJar() future = time2netscape(time.time()+3600) - with test.support.check_no_warnings(self): + with warnings_helper.check_no_warnings(self): headers = [f"Set-Cookie: FOO=BAR; path=/; expires={future}"] req = urllib.request.Request("http://www.coyote.com/") res = FakeResponse(headers, "http://www.coyote.com/") @@ -1713,7 +1715,7 @@ def test_rejection(self): self.assertEqual(len(c), 6) # save and restore - filename = test.support.TESTFN + filename = os_helper.TESTFN try: c.save(filename, ignore_discard=True) @@ -1753,7 +1755,7 @@ def test_mozilla(self): # Save / load Mozilla/Netscape cookie file format. year_plus_one = time.localtime()[0] + 1 - filename = test.support.TESTFN + filename = os_helper.TESTFN c = MozillaCookieJar(filename, policy=DefaultCookiePolicy(rfc2965=True)) diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py index 0c871afca37bd..a7e1719ab60ee 100644 --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -67,7 +67,7 @@ def stop(self): class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = threading_helper.threading_setup() - os.environ = support.EnvironmentVarGuard() + os.environ = os_helper.EnvironmentVarGuard() self.server_started = threading.Event() self.thread = TestServerThread(self, self.request_handler) self.thread.start() @@ -621,7 +621,7 @@ def setUp(self): # The shebang line should be pure ASCII: use symlink if possible. # See issue #7668. self._pythonexe_symlink = None - if support.can_symlink(): + if os_helper.can_symlink(): self.pythonexe = os.path.join(self.parent_dir, 'python') self._pythonexe_symlink = support.PythonSymlink(self.pythonexe).__enter__() else: diff --git a/Lib/test/test_import/__init__.py b/Lib/test/test_import/__init__.py index f4a83d2e7a13a..6fd3983a20f68 100644 --- a/Lib/test/test_import/__init__.py +++ b/Lib/test/test_import/__init__.py @@ -18,7 +18,6 @@ import unittest from unittest import mock -import test.support from test.support import os_helper from test.support import (is_jython, swap_attr, swap_item, cpython_only) from test.support.import_helper import ( @@ -480,7 +479,7 @@ def test_dll_dependency_import(self): os.path.dirname(pydname), "sqlite3{}.dll".format("_d" if "_d" in pydname else "")) - with test.support.temp_dir() as tmp: + with os_helper.temp_dir() as tmp: tmp2 = os.path.join(tmp, "DLLs") os.mkdir(tmp2) diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py index 69c44710f0b5a..0d84ff8bce2c0 100644 --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -285,7 +285,7 @@ def test_realpath_relative(self): def test_realpath_broken_symlinks(self): ABSTFN = ntpath.abspath(os_helper.TESTFN) os.mkdir(ABSTFN) - self.addCleanup(support.rmtree, ABSTFN) + self.addCleanup(os_helper.rmtree, ABSTFN) with support.change_cwd(ABSTFN): os.mkdir("subdir") @@ -427,9 +427,9 @@ def test_realpath_cwd(self): ABSTFN = ntpath.abspath(os_helper.TESTFN) os_helper.unlink(ABSTFN) - support.rmtree(ABSTFN) + os_helper.rmtree(ABSTFN) os.mkdir(ABSTFN) - self.addCleanup(support.rmtree, ABSTFN) + self.addCleanup(os_helper.rmtree, ABSTFN) test_dir_long = ntpath.join(ABSTFN, "MyVeryLongDirectoryName") os.mkdir(test_dir_long) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py index 5901939725e18..97b5c5de95bbc 100644 --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -7,6 +7,7 @@ import unittest import test.support from test import support +from test.support import os_helper from test.support import socket_helper from test.support import captured_stderr from test.support.os_helper import TESTFN, EnvironmentVarGuard, change_cwd @@ -601,7 +602,7 @@ class _pthFileTests(unittest.TestCase): def _create_underpth_exe(self, lines, exe_pth=True): import _winapi temp_dir = tempfile.mkdtemp() - self.addCleanup(test.support.rmtree, temp_dir) + self.addCleanup(os_helper.rmtree, temp_dir) exe_file = os.path.join(temp_dir, os.path.split(sys.executable)[1]) dll_src_file = _winapi.GetModuleFileName(sys.dllhandle) dll_file = os.path.join(temp_dir, os.path.split(dll_src_file)[1]) diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py index 5db8cec567afb..7cdd115a95153 100644 --- a/Lib/test/test_socketserver.py +++ b/Lib/test/test_socketserver.py @@ -15,6 +15,7 @@ import test.support from test.support import reap_children, verbose +from test.support import os_helper from test.support import socket_helper from test.support import threading_helper @@ -299,7 +300,7 @@ class ErrorHandlerTest(unittest.TestCase): KeyboardInterrupt are not passed.""" def tearDown(self): - test.support.unlink(test.support.TESTFN) + os_helper.unlink(os_helper.TESTFN) def test_sync_handled(self): BaseErrorTestServer(ValueError) @@ -329,7 +330,7 @@ def test_forking_not_handled(self): self.check_result(handled=False) def check_result(self, handled): - with open(test.support.TESTFN) as log: + with open(os_helper.TESTFN) as log: expected = 'Handler called\n' + 'Error handled\n' * handled self.assertEqual(log.read(), expected) @@ -347,7 +348,7 @@ def __init__(self, exception): self.wait_done() def handle_error(self, request, client_address): - with open(test.support.TESTFN, 'a') as log: + with open(os_helper.TESTFN, 'a') as log: log.write('Error handled\n') def wait_done(self): @@ -356,7 +357,7 @@ def wait_done(self): class BadHandler(socketserver.BaseRequestHandler): def handle(self): - with open(test.support.TESTFN, 'a') as log: + with open(os_helper.TESTFN, 'a') as log: log.write('Handler called\n') raise self.server.exception('Test error') diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py index bf415dda557e6..997110732a176 100644 --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -15,10 +15,10 @@ import sys import unittest from test import support +from test.support import import_helper from decimal import Decimal from fractions import Fraction -from test import support # Module to be tested. @@ -179,8 +179,10 @@ class _DoNothing: # We prefer this for testing numeric values that may not be exactly equal, # and avoid using TestCase.assertAlmostEqual, because it sucks :-) -py_statistics = support.import_fresh_module('statistics', blocked=['_statistics']) -c_statistics = support.import_fresh_module('statistics', fresh=['_statistics']) +py_statistics = import_helper.import_fresh_module('statistics', + blocked=['_statistics']) +c_statistics = import_helper.import_fresh_module('statistics', + fresh=['_statistics']) class TestModules(unittest.TestCase): diff --git a/Lib/test/test_tabnanny.py b/Lib/test/test_tabnanny.py index 95840d6ac0c5f..4dfbd2985d5b5 100644 --- a/Lib/test/test_tabnanny.py +++ b/Lib/test/test_tabnanny.py @@ -12,7 +12,8 @@ import tempfile import textwrap from test.support import (captured_stderr, captured_stdout, script_helper, - findfile, unlink) + findfile) +from test.support.os_helper import unlink SOURCE_CODES = { diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index db982dac8d653..cd2a30e533ae0 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -6,6 +6,7 @@ import warnings from test import support from test.support import import_helper +from test.support import os_helper # Skip this test if the _tkinter module wasn't built. _tkinter = import_helper.import_module('_tkinter') @@ -192,26 +193,26 @@ def test_getboolean(self): def testEvalFile(self): tcl = self.interp - with open(support.TESTFN, 'w') as f: - self.addCleanup(support.unlink, support.TESTFN) + with open(os_helper.TESTFN, 'w') as f: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) f.write("""set a 1 set b 2 set c [ expr $a + $b ] """) - tcl.evalfile(support.TESTFN) + tcl.evalfile(os_helper.TESTFN) self.assertEqual(tcl.eval('set a'),'1') self.assertEqual(tcl.eval('set b'),'2') self.assertEqual(tcl.eval('set c'),'3') def test_evalfile_null_in_result(self): tcl = self.interp - with open(support.TESTFN, 'w') as f: - self.addCleanup(support.unlink, support.TESTFN) + with open(os_helper.TESTFN, 'w') as f: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) f.write(""" set a "a\0b" set b "a\\0b" """) - tcl.evalfile(support.TESTFN) + tcl.evalfile(os_helper.TESTFN) self.assertEqual(tcl.eval('set a'), 'a\x00b') self.assertEqual(tcl.eval('set b'), 'a\x00b') @@ -243,7 +244,7 @@ def testLoadWithUNC(self): if not os.path.exists(unc_name): raise unittest.SkipTest('Cannot connect to UNC Path') - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.unset("TCL_LIBRARY") stdout = subprocess.check_output( [unc_name, '-c', 'import tkinter; print(tkinter)']) diff --git a/Lib/test/test_tools/test_md5sum.py b/Lib/test/test_tools/test_md5sum.py index 321bc4bb36282..bfc1f287fff6e 100644 --- a/Lib/test/test_tools/test_md5sum.py +++ b/Lib/test/test_tools/test_md5sum.py @@ -2,7 +2,7 @@ import os import unittest -from test import support +from test.support import os_helper from test.support import hashlib_helper from test.support.script_helper import assert_python_ok, assert_python_failure @@ -15,8 +15,8 @@ class MD5SumTests(unittest.TestCase): @classmethod def setUpClass(cls): cls.script = os.path.join(scriptsdir, 'md5sum.py') - os.mkdir(support.TESTFN) - cls.fodder = os.path.join(support.TESTFN, 'md5sum.fodder') + os.mkdir(os_helper.TESTFN) + cls.fodder = os.path.join(os_helper.TESTFN, 'md5sum.fodder') with open(cls.fodder, 'wb') as f: f.write(b'md5sum\r\ntest file\r\n') cls.fodder_md5 = b'd38dae2eb1ab346a292ef6850f9e1a0d' @@ -24,7 +24,7 @@ def setUpClass(cls): @classmethod def tearDownClass(cls): - support.rmtree(support.TESTFN) + os_helper.rmtree(os_helper.TESTFN) def test_noargs(self): rc, out, err = assert_python_ok(self.script) diff --git a/Lib/test/test_turtle.py b/Lib/test/test_turtle.py index 39b3d96fb43bb..46ff4a3aac709 100644 --- a/Lib/test/test_turtle.py +++ b/Lib/test/test_turtle.py @@ -2,6 +2,7 @@ import unittest from test import support from test.support import import_helper +from test.support import os_helper turtle = import_helper.import_module('turtle') @@ -52,10 +53,10 @@ class TurtleConfigTest(unittest.TestCase): def get_cfg_file(self, cfg_str): - self.addCleanup(support.unlink, support.TESTFN) - with open(support.TESTFN, 'w') as f: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + with open(os_helper.TESTFN, 'w') as f: f.write(cfg_str) - return support.TESTFN + return os_helper.TESTFN def test_config_dict(self): diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py index cb74685715d35..c1d55ee8b29b3 100644 --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -1,6 +1,7 @@ import errno import unittest from test import support +from test.support import os_helper from test.support import socket_helper from test.test_urllib2 import sanepathname2url @@ -148,7 +149,7 @@ def test_ftp(self): self._test_urls(urls, self._extra_handlers()) def test_file(self): - TESTFN = support.TESTFN + TESTFN = os_helper.TESTFN f = open(TESTFN, 'w') try: f.write('hi there\n') diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py index 28680aa6b2405..773101ce41f60 100644 --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -1,5 +1,6 @@ import unittest from test import support +from test.support import os_helper from test.support import socket_helper import contextlib @@ -162,7 +163,7 @@ def urlretrieve(self, *args, **kwargs): try: yield file_location, info finally: - support.unlink(file_location) + os_helper.unlink(file_location) def test_basic(self): # Test basic functionality. @@ -176,8 +177,8 @@ def test_basic(self): def test_specified_path(self): # Make sure that specifying the location of the file to write to works. with self.urlretrieve(self.logo, - support.TESTFN) as (file_location, info): - self.assertEqual(file_location, support.TESTFN) + os_helper.TESTFN) as (file_location, info): + self.assertEqual(file_location, os_helper.TESTFN) self.assertTrue(os.path.exists(file_location)) with open(file_location, 'rb') as f: self.assertTrue(f.read(), "reading from temporary file failed") diff --git a/Lib/test/test_webbrowser.py b/Lib/test/test_webbrowser.py index 6ceb49069f656..673cc995d3f5a 100644 --- a/Lib/test/test_webbrowser.py +++ b/Lib/test/test_webbrowser.py @@ -6,6 +6,7 @@ from unittest import mock from test import support from test.support import import_helper +from test.support import os_helper URL = 'http://www.example.com' @@ -305,7 +306,7 @@ def test_environment(self): browser = webbrowser.get().name except (webbrowser.Error, AttributeError) as err: self.skipTest(str(err)) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["BROWSER"] = browser webbrowser = import_helper.import_fresh_module('webbrowser') webbrowser.get() @@ -318,12 +319,12 @@ def test_environment_preferred(self): except (webbrowser.Error, AttributeError, IndexError) as err: self.skipTest(str(err)) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["BROWSER"] = least_preferred_browser webbrowser = import_helper.import_fresh_module('webbrowser') self.assertEqual(webbrowser.get().name, least_preferred_browser) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env["BROWSER"] = sys.executable webbrowser = import_helper.import_fresh_module('webbrowser') self.assertEqual(webbrowser.get().name, sys.executable) diff --git a/Lib/test/test_winsound.py b/Lib/test/test_winsound.py index dca77cb45bfa9..3c3359b9004fd 100644 --- a/Lib/test/test_winsound.py +++ b/Lib/test/test_winsound.py @@ -5,9 +5,11 @@ import unittest from test import support +from test.support import import_helper + support.requires('audio') -winsound = support.import_module('winsound') +winsound = import_helper.import_module('winsound') # Unless we actually have an ear in the room, we have no idea whether a sound diff --git a/Lib/tkinter/test/test_tkinter/test_images.py b/Lib/tkinter/test/test_tkinter/test_images.py index 2805d35a1f5b1..6c6cb4e148573 100644 --- a/Lib/tkinter/test/test_tkinter/test_images.py +++ b/Lib/tkinter/test/test_tkinter/test_images.py @@ -1,6 +1,7 @@ import unittest import tkinter from test import support +from test.support import os_helper from tkinter.test.support import AbstractTkTest, requires_tcl support.requires('gui') @@ -296,12 +297,12 @@ def test_get(self): def test_write(self): image = self.create() - self.addCleanup(support.unlink, support.TESTFN) + self.addCleanup(os_helper.unlink, os_helper.TESTFN) - image.write(support.TESTFN) + image.write(os_helper.TESTFN) image2 = tkinter.PhotoImage('::img::test2', master=self.root, format='ppm', - file=support.TESTFN) + file=os_helper.TESTFN) self.assertEqual(str(image2), '::img::test2') self.assertEqual(image2.type(), 'photo') self.assertEqual(image2.width(), 16) @@ -309,10 +310,10 @@ def test_write(self): self.assertEqual(image2.get(0, 0), image.get(0, 0)) self.assertEqual(image2.get(15, 8), image.get(15, 8)) - image.write(support.TESTFN, format='gif', from_coords=(4, 6, 6, 9)) + image.write(os_helper.TESTFN, format='gif', from_coords=(4, 6, 6, 9)) image3 = tkinter.PhotoImage('::img::test3', master=self.root, format='gif', - file=support.TESTFN) + file=os_helper.TESTFN) self.assertEqual(str(image3), '::img::test3') self.assertEqual(image3.type(), 'photo') self.assertEqual(image3.width(), 2) diff --git a/Lib/tkinter/test/test_tkinter/test_loadtk.py b/Lib/tkinter/test/test_tkinter/test_loadtk.py index bab7bcd37ce31..760ba72134082 100644 --- a/Lib/tkinter/test/test_tkinter/test_loadtk.py +++ b/Lib/tkinter/test/test_tkinter/test_loadtk.py @@ -2,6 +2,7 @@ import sys import unittest import test.support as test_support +from test.support import os_helper from tkinter import Tcl, TclError test_support.requires('gui') @@ -24,7 +25,7 @@ def testLoadTkFailure(self): # XXX Maybe on tk older than 8.4.13 it would be possible, # see tkinter.h. return - with test_support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: if 'DISPLAY' in os.environ: del env['DISPLAY'] # on some platforms, deleting environment variables From webhook-mailer at python.org Thu Aug 6 12:36:30 2020 From: webhook-mailer at python.org (Steve Dower) Date: Thu, 06 Aug 2020 16:36:30 -0000 Subject: [Python-checkins] bpo-41492: Fixes the description appearing in UAC prompts on Windows (GH-21754) Message-ID: https://github.com/python/cpython/commit/777b611c8c5676b80898a429f71d28e59bddc49d commit: 777b611c8c5676b80898a429f71d28e59bddc49d branch: master author: Steve Dower committer: GitHub date: 2020-08-06T17:36:22+01:00 summary: bpo-41492: Fixes the description appearing in UAC prompts on Windows (GH-21754) files: A Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst M .azure-pipelines/windows-release/msi-steps.yml M .azure-pipelines/windows-release/stage-pack-msix.yml M .azure-pipelines/windows-release/stage-sign.yml diff --git a/.azure-pipelines/windows-release/msi-steps.yml b/.azure-pipelines/windows-release/msi-steps.yml index a460eb1bac8fe..307510a40dd4e 100644 --- a/.azure-pipelines/windows-release/msi-steps.yml +++ b/.azure-pipelines/windows-release/msi-steps.yml @@ -1,6 +1,12 @@ steps: - template: ./checkout.yml + - powershell: | + $d = (.\PCbuild\build.bat -V) | %{ if($_ -match '\s+(\w+):\s*(.+)\s*$') { @{$Matches[1] = $Matches[2];} }}; + Write-Host "##vso[task.setvariable variable=SigningDescription]Python $($d.PythonVersion)" + displayName: 'Update signing description' + condition: and(succeeded(), not(variables['SigningDescription'])) + - task: DownloadPipelineArtifact at 1 displayName: 'Download artifact: doc' inputs: diff --git a/.azure-pipelines/windows-release/stage-pack-msix.yml b/.azure-pipelines/windows-release/stage-pack-msix.yml index 07e343a0b4e0c..26a5712e845ca 100644 --- a/.azure-pipelines/windows-release/stage-pack-msix.yml +++ b/.azure-pipelines/windows-release/stage-pack-msix.yml @@ -105,9 +105,15 @@ jobs: clean: all steps: - - checkout: none + - template: ./checkout.yml - template: ./find-sdk.yml + - powershell: | + $d = (.\PCbuild\build.bat -V) | %{ if($_ -match '\s+(\w+):\s*(.+)\s*$') { @{$Matches[1] = $Matches[2];} }}; + Write-Host "##vso[task.setvariable variable=SigningDescription]Python $($d.PythonVersion)" + displayName: 'Update signing description' + condition: and(succeeded(), not(variables['SigningDescription'])) + - task: DownloadBuildArtifacts at 0 displayName: 'Download Artifact: unsigned_msix' inputs: diff --git a/.azure-pipelines/windows-release/stage-sign.yml b/.azure-pipelines/windows-release/stage-sign.yml index 4d757ae8fca03..584772af8b428 100644 --- a/.azure-pipelines/windows-release/stage-sign.yml +++ b/.azure-pipelines/windows-release/stage-sign.yml @@ -26,6 +26,12 @@ jobs: - template: ./checkout.yml - template: ./find-sdk.yml + - powershell: | + $d = (.\PCbuild\build.bat -V) | %{ if($_ -match '\s+(\w+):\s*(.+)\s*$') { @{$Matches[1] = $Matches[2];} }}; + Write-Host "##vso[task.setvariable variable=SigningDescription]Python $($d.PythonVersion)" + displayName: 'Update signing description' + condition: and(succeeded(), not(variables['SigningDescription'])) + - powershell: | Write-Host "##vso[build.addbuildtag]signed" displayName: 'Add build tags' diff --git a/Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst b/Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst new file mode 100644 index 0000000000000..065803e2c2075 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst @@ -0,0 +1 @@ +Fixes the description that appears in UAC prompts. From webhook-mailer at python.org Thu Aug 6 12:53:08 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Thu, 06 Aug 2020 16:53:08 -0000 Subject: [Python-checkins] bpo-41492: Fixes the description appearing in UAC prompts on Windows (GH-21754) Message-ID: https://github.com/python/cpython/commit/713ba03276c4ddc33c9debc51b03164ea18eead6 commit: 713ba03276c4ddc33c9debc51b03164ea18eead6 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-06T09:52:57-07:00 summary: bpo-41492: Fixes the description appearing in UAC prompts on Windows (GH-21754) (cherry picked from commit 777b611c8c5676b80898a429f71d28e59bddc49d) Co-authored-by: Steve Dower files: A Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst M .azure-pipelines/windows-release/msi-steps.yml M .azure-pipelines/windows-release/stage-pack-msix.yml M .azure-pipelines/windows-release/stage-sign.yml diff --git a/.azure-pipelines/windows-release/msi-steps.yml b/.azure-pipelines/windows-release/msi-steps.yml index a460eb1bac8fe..307510a40dd4e 100644 --- a/.azure-pipelines/windows-release/msi-steps.yml +++ b/.azure-pipelines/windows-release/msi-steps.yml @@ -1,6 +1,12 @@ steps: - template: ./checkout.yml + - powershell: | + $d = (.\PCbuild\build.bat -V) | %{ if($_ -match '\s+(\w+):\s*(.+)\s*$') { @{$Matches[1] = $Matches[2];} }}; + Write-Host "##vso[task.setvariable variable=SigningDescription]Python $($d.PythonVersion)" + displayName: 'Update signing description' + condition: and(succeeded(), not(variables['SigningDescription'])) + - task: DownloadPipelineArtifact at 1 displayName: 'Download artifact: doc' inputs: diff --git a/.azure-pipelines/windows-release/stage-pack-msix.yml b/.azure-pipelines/windows-release/stage-pack-msix.yml index 07e343a0b4e0c..26a5712e845ca 100644 --- a/.azure-pipelines/windows-release/stage-pack-msix.yml +++ b/.azure-pipelines/windows-release/stage-pack-msix.yml @@ -105,9 +105,15 @@ jobs: clean: all steps: - - checkout: none + - template: ./checkout.yml - template: ./find-sdk.yml + - powershell: | + $d = (.\PCbuild\build.bat -V) | %{ if($_ -match '\s+(\w+):\s*(.+)\s*$') { @{$Matches[1] = $Matches[2];} }}; + Write-Host "##vso[task.setvariable variable=SigningDescription]Python $($d.PythonVersion)" + displayName: 'Update signing description' + condition: and(succeeded(), not(variables['SigningDescription'])) + - task: DownloadBuildArtifacts at 0 displayName: 'Download Artifact: unsigned_msix' inputs: diff --git a/.azure-pipelines/windows-release/stage-sign.yml b/.azure-pipelines/windows-release/stage-sign.yml index 4d757ae8fca03..584772af8b428 100644 --- a/.azure-pipelines/windows-release/stage-sign.yml +++ b/.azure-pipelines/windows-release/stage-sign.yml @@ -26,6 +26,12 @@ jobs: - template: ./checkout.yml - template: ./find-sdk.yml + - powershell: | + $d = (.\PCbuild\build.bat -V) | %{ if($_ -match '\s+(\w+):\s*(.+)\s*$') { @{$Matches[1] = $Matches[2];} }}; + Write-Host "##vso[task.setvariable variable=SigningDescription]Python $($d.PythonVersion)" + displayName: 'Update signing description' + condition: and(succeeded(), not(variables['SigningDescription'])) + - powershell: | Write-Host "##vso[build.addbuildtag]signed" displayName: 'Add build tags' diff --git a/Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst b/Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst new file mode 100644 index 0000000000000..065803e2c2075 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-08-06-16-59-10.bpo-41492.2FQ9cM.rst @@ -0,0 +1 @@ +Fixes the description that appears in UAC prompts. From webhook-mailer at python.org Thu Aug 6 18:09:56 2020 From: webhook-mailer at python.org (Nathan M) Date: Thu, 06 Aug 2020 22:09:56 -0000 Subject: [Python-checkins] bpo-41371: Handle lzma lib import error in test_zoneinfo.py (GH-21734) Message-ID: https://github.com/python/cpython/commit/5f0769a7529fcbc28190864f098f192053a10747 commit: 5f0769a7529fcbc28190864f098f192053a10747 branch: master author: Nathan M committer: GitHub date: 2020-08-06T15:09:40-07:00 summary: bpo-41371: Handle lzma lib import error in test_zoneinfo.py (GH-21734) files: M Lib/test/test_zoneinfo/test_zoneinfo.py diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index 1f1fa60f1ffc1..d16e0d2c33106 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -6,7 +6,6 @@ import importlib.metadata import io import json -import lzma import os import pathlib import pickle @@ -20,7 +19,9 @@ from . import _support as test_support from ._support import OS_ENV_LOCK, TZPATH_TEST_LOCK, ZoneInfoTestBase +from test.support.import_helper import import_module +lzma = import_module('lzma') py_zoneinfo, c_zoneinfo = test_support.get_modules() try: From webhook-mailer at python.org Fri Aug 7 01:08:59 2020 From: webhook-mailer at python.org (Inada Naoki) Date: Fri, 07 Aug 2020 05:08:59 -0000 Subject: [Python-checkins] bpo-41493: Refactoring dictresize (GH-21751) Message-ID: https://github.com/python/cpython/commit/d9323a8c6e07071a59dc4c910661db33236c01b2 commit: d9323a8c6e07071a59dc4c910661db33236c01b2 branch: master author: Inada Naoki committer: GitHub date: 2020-08-07T14:08:55+09:00 summary: bpo-41493: Refactoring dictresize (GH-21751) Split newsize calculation into new function. dictresize() now accepts exact newsize. files: M Objects/dictobject.c diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 1b7ae06d82271..6c3fc62d2ecc7 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -111,6 +111,7 @@ converting the dict to the combined table. #define PyDict_MINSIZE 8 #include "Python.h" +#include "pycore_bitutils.h" // _Py_bit_length #include "pycore_gc.h" // _PyObject_GC_IS_TRACKED() #include "pycore_object.h" // _PyObject_GC_TRACK() #include "pycore_pyerrors.h" // _PyErr_Fetch() @@ -236,7 +237,7 @@ lookdict_unicode_nodummy(PyDictObject *mp, PyObject *key, static Py_ssize_t lookdict_split(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject **value_addr); -static int dictresize(PyDictObject *mp, Py_ssize_t minused); +static int dictresize(PyDictObject *mp, Py_ssize_t newsize); static PyObject* dict_iter(PyDictObject *dict); @@ -411,18 +412,40 @@ dictkeys_set_index(PyDictKeysObject *keys, Py_ssize_t i, Py_ssize_t ix) */ #define USABLE_FRACTION(n) (((n) << 1)/3) -/* ESTIMATE_SIZE is reverse function of USABLE_FRACTION. +/* Find the smallest dk_size >= minsize. */ +static inline Py_ssize_t +calculate_keysize(Py_ssize_t minsize) +{ +#if SIZEOF_LONG == SIZEOF_SIZE_T + minsize = (minsize | PyDict_MINSIZE) - 1; + return 1LL << _Py_bit_length(minsize | (PyDict_MINSIZE-1)); +#elif defined(_MSC_VER) + // On 64bit Windows, sizeof(long) == 4. + minsize = (minsize | PyDict_MINSIZE) - 1; + unsigned long msb; + _BitScanReverse64(&msb, (uint64_t)minsize); + return 1LL << (msb + 1); +#else + Py_ssize_t size; + for (size = PyDict_MINSIZE; + size < minsize && size > 0; + size <<= 1) + ; + return size; +#endif +} + +/* estimate_keysize is reverse function of USABLE_FRACTION. + * * This can be used to reserve enough size to insert n entries without * resizing. */ -#define ESTIMATE_SIZE(n) (((n)*3+1) >> 1) +static inline Py_ssize_t +estimate_keysize(Py_ssize_t n) +{ + return calculate_keysize((n*3 + 1) / 2); +} -/* Alternative fraction that is otherwise close enough to 2n/3 to make - * little difference. 8 * 2/3 == 8 * 5/8 == 5. 16 * 2/3 == 16 * 5/8 == 10. - * 32 * 2/3 = 21, 32 * 5/8 = 20. - * Its advantage is that it is faster to compute on machines with slow division. - * #define USABLE_FRACTION(n) (((n) >> 1) + ((n) >> 2) - ((n) >> 3)) - */ /* GROWTH_RATE. Growth rate upon hitting maximum load. * Currently set to used*3. @@ -1036,7 +1059,7 @@ find_empty_slot(PyDictKeysObject *keys, Py_hash_t hash) static int insertion_resize(PyDictObject *mp) { - return dictresize(mp, GROWTH_RATE(mp)); + return dictresize(mp, calculate_keysize(GROWTH_RATE(mp))); } /* @@ -1194,22 +1217,19 @@ After resizing a table is always combined, but can be resplit by make_keys_shared(). */ static int -dictresize(PyDictObject *mp, Py_ssize_t minsize) +dictresize(PyDictObject *mp, Py_ssize_t newsize) { - Py_ssize_t newsize, numentries; + Py_ssize_t numentries; PyDictKeysObject *oldkeys; PyObject **oldvalues; PyDictKeyEntry *oldentries, *newentries; - /* Find the smallest table size > minused. */ - for (newsize = PyDict_MINSIZE; - newsize < minsize && newsize > 0; - newsize <<= 1) - ; if (newsize <= 0) { PyErr_NoMemory(); return -1; } + assert(IS_POWER_OF_2(newsize)); + assert(newsize >= PyDict_MINSIZE); oldkeys = mp->ma_keys; @@ -1355,13 +1375,8 @@ _PyDict_NewPresized(Py_ssize_t minused) newsize = max_presize; } else { - Py_ssize_t minsize = ESTIMATE_SIZE(minused); - newsize = PyDict_MINSIZE*2; - while (newsize < minsize) { - newsize <<= 1; - } + newsize = estimate_keysize(minused); } - assert(IS_POWER_OF_2(newsize)); new_keys = new_keys_object(newsize); if (new_keys == NULL) @@ -1930,7 +1945,7 @@ _PyDict_FromKeys(PyObject *cls, PyObject *iterable, PyObject *value) PyObject *key; Py_hash_t hash; - if (dictresize(mp, ESTIMATE_SIZE(PyDict_GET_SIZE(iterable)))) { + if (dictresize(mp, estimate_keysize(PyDict_GET_SIZE(iterable)))) { Py_DECREF(d); return NULL; } @@ -1949,7 +1964,7 @@ _PyDict_FromKeys(PyObject *cls, PyObject *iterable, PyObject *value) PyObject *key; Py_hash_t hash; - if (dictresize(mp, ESTIMATE_SIZE(PySet_GET_SIZE(iterable)))) { + if (dictresize(mp, estimate_keysize(PySet_GET_SIZE(iterable)))) { Py_DECREF(d); return NULL; } @@ -2558,7 +2573,7 @@ dict_merge(PyObject *a, PyObject *b, int override) * that there will be no (or few) overlapping keys. */ if (USABLE_FRACTION(mp->ma_keys->dk_size) < other->ma_used) { - if (dictresize(mp, ESTIMATE_SIZE(mp->ma_used + other->ma_used))) { + if (dictresize(mp, estimate_keysize(mp->ma_used + other->ma_used))) { return -1; } } From webhook-mailer at python.org Fri Aug 7 01:22:01 2020 From: webhook-mailer at python.org (pxinwr) Date: Fri, 07 Aug 2020 05:22:01 -0000 Subject: [Python-checkins] bpo-41440: add os.cpu_count() support for VxWorks RTOS (GH-21685) Message-ID: https://github.com/python/cpython/commit/3405e0542839cde94df9833b3809710a67a33c7c commit: 3405e0542839cde94df9833b3809710a67a33c7c branch: master author: pxinwr committer: GitHub date: 2020-08-07T14:21:52+09:00 summary: bpo-41440: add os.cpu_count() support for VxWorks RTOS (GH-21685) files: A Misc/NEWS.d/next/Library/2020-07-30-14-56-58.bpo-41440.rju34k.rst M Doc/whatsnew/3.10.rst M Modules/posixmodule.c diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index ec0343f2ce71e..62bb1438416e6 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -120,6 +120,12 @@ Added the *root_dir* and *dir_fd* parameters in :func:`~glob.glob` and :func:`~glob.iglob` which allow to specify the root directory for searching. (Contributed by Serhiy Storchaka in :issue:`38144`.) +os +-- + +Added :func:`os.cpu_count()` support for VxWorks RTOS. +(Contributed by Peixing Xin in :issue:`41440`.) + py_compile ---------- diff --git a/Misc/NEWS.d/next/Library/2020-07-30-14-56-58.bpo-41440.rju34k.rst b/Misc/NEWS.d/next/Library/2020-07-30-14-56-58.bpo-41440.rju34k.rst new file mode 100644 index 0000000000000..3ee1f656d1870 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-07-30-14-56-58.bpo-41440.rju34k.rst @@ -0,0 +1 @@ +Add :func:`os.cpu_count()` support for VxWorks RTOS. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index efd99544f5a99..a6a4b9f012f00 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -32,6 +32,9 @@ # include #endif +#ifdef __VXWORKS__ +# include "pycore_bitutils.h" // _Py_popcount32() +#endif #include "pycore_ceval.h" // _PyEval_ReInitThreads() #include "pycore_import.h" // _PyImport_ReInitLock() #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() @@ -12607,6 +12610,8 @@ os_cpu_count_impl(PyObject *module) ncpu = mpctl(MPC_GETNUMSPUS, NULL, NULL); #elif defined(HAVE_SYSCONF) && defined(_SC_NPROCESSORS_ONLN) ncpu = sysconf(_SC_NPROCESSORS_ONLN); +#elif defined(__VXWORKS__) + ncpu = _Py_popcount32(vxCpuEnabledGet()); #elif defined(__DragonFly__) || \ defined(__OpenBSD__) || \ defined(__FreeBSD__) || \ From webhook-mailer at python.org Fri Aug 7 01:38:53 2020 From: webhook-mailer at python.org (Zackery Spytz) Date: Fri, 07 Aug 2020 05:38:53 -0000 Subject: [Python-checkins] bpo-39871: Fix an error in a news entry (GH-21749) Message-ID: https://github.com/python/cpython/commit/54636355805dd2877bb54fbad8d967e1ddd8b553 commit: 54636355805dd2877bb54fbad8d967e1ddd8b553 branch: master author: Zackery Spytz committer: GitHub date: 2020-08-07T14:38:48+09:00 summary: bpo-39871: Fix an error in a news entry (GH-21749) files: M Misc/NEWS.d/3.9.0a5.rst diff --git a/Misc/NEWS.d/3.9.0a5.rst b/Misc/NEWS.d/3.9.0a5.rst index 39e017768c3ad..355a3fc22350c 100644 --- a/Misc/NEWS.d/3.9.0a5.rst +++ b/Misc/NEWS.d/3.9.0a5.rst @@ -232,7 +232,7 @@ exits before trying to take the GIL. Fix a possible :exc:`SystemError` in ``math.{atan2,copysign,remainder}()`` when the first argument cannot be converted to a :class:`float`. Patch by -Zachary Spytz. +Zackery Spytz. .. From webhook-mailer at python.org Fri Aug 7 03:32:07 2020 From: webhook-mailer at python.org (Inada Naoki) Date: Fri, 07 Aug 2020 07:32:07 -0000 Subject: [Python-checkins] bpo-41098: Doc: Add missing deprecated directives (GH-21162) Message-ID: https://github.com/python/cpython/commit/46e19b61d31ba99f049258efa4ff1334856a3643 commit: 46e19b61d31ba99f049258efa4ff1334856a3643 branch: master author: Inada Naoki committer: GitHub date: 2020-08-07T16:31:53+09:00 summary: bpo-41098: Doc: Add missing deprecated directives (GH-21162) PyUnicodeEncodeError_Create has been deprecated with `Py_DEPRECATED` macro. But it was not documented. files: M Doc/c-api/exceptions.rst M Include/cpython/pyerrors.h diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index e7805ba143c58..b4722ff81979f 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -637,11 +637,21 @@ The following functions are used to create and modify Unicode exceptions from C. *object*, *length*, *start*, *end* and *reason*. *encoding* and *reason* are UTF-8 encoded strings. + .. deprecated:: 3.3 3.11 + + ``Py_UNICODE`` is deprecated since Python 3.3. Please migrate to + ``PyObject_CallFunction(PyExc_UnicodeEncodeError, "sOnns", ...)``. + .. c:function:: PyObject* PyUnicodeTranslateError_Create(const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) Create a :class:`UnicodeTranslateError` object with the attributes *object*, *length*, *start*, *end* and *reason*. *reason* is a UTF-8 encoded string. + .. deprecated:: 3.3 3.11 + + ``Py_UNICODE`` is deprecated since Python 3.3. Please migrate to + ``PyObject_CallFunction(PyExc_UnicodeTranslateError, "Onns", ...)``. + .. c:function:: PyObject* PyUnicodeDecodeError_GetEncoding(PyObject *exc) PyObject* PyUnicodeEncodeError_GetEncoding(PyObject *exc) diff --git a/Include/cpython/pyerrors.h b/Include/cpython/pyerrors.h index 3f347dc2e2d62..c2500d927bf7f 100644 --- a/Include/cpython/pyerrors.h +++ b/Include/cpython/pyerrors.h @@ -145,7 +145,10 @@ PyAPI_FUNC(PyObject *) PyErr_ProgramTextObject( PyObject *filename, int lineno); -/* Create a UnicodeEncodeError object */ +/* Create a UnicodeEncodeError object. + * + * TODO: This API will be removed in Python 3.11. + */ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject *) PyUnicodeEncodeError_Create( const char *encoding, /* UTF-8 encoded string */ const Py_UNICODE *object, @@ -155,7 +158,10 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject *) PyUnicodeEncodeError_Create( const char *reason /* UTF-8 encoded string */ ); -/* Create a UnicodeTranslateError object */ +/* Create a UnicodeTranslateError object. + * + * TODO: This API will be removed in Python 3.11. + */ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject *) PyUnicodeTranslateError_Create( const Py_UNICODE *object, Py_ssize_t length, From webhook-mailer at python.org Fri Aug 7 03:49:58 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 07 Aug 2020 07:49:58 -0000 Subject: [Python-checkins] bpo-41098: Doc: Add missing deprecated directives (GH-21162) Message-ID: https://github.com/python/cpython/commit/dc98a5468a48ecffe95a6c3a07ce21366477bdd7 commit: dc98a5468a48ecffe95a6c3a07ce21366477bdd7 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-07T00:49:46-07:00 summary: bpo-41098: Doc: Add missing deprecated directives (GH-21162) PyUnicodeEncodeError_Create has been deprecated with `Py_DEPRECATED` macro. But it was not documented. (cherry picked from commit 46e19b61d31ba99f049258efa4ff1334856a3643) Co-authored-by: Inada Naoki files: M Doc/c-api/exceptions.rst M Include/cpython/pyerrors.h diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index 85a3d941ed8c1..bbc1ee28d9b08 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -635,11 +635,21 @@ The following functions are used to create and modify Unicode exceptions from C. *object*, *length*, *start*, *end* and *reason*. *encoding* and *reason* are UTF-8 encoded strings. + .. deprecated:: 3.3 3.11 + + ``Py_UNICODE`` is deprecated since Python 3.3. Please migrate to + ``PyObject_CallFunction(PyExc_UnicodeEncodeError, "sOnns", ...)``. + .. c:function:: PyObject* PyUnicodeTranslateError_Create(const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) Create a :class:`UnicodeTranslateError` object with the attributes *object*, *length*, *start*, *end* and *reason*. *reason* is a UTF-8 encoded string. + .. deprecated:: 3.3 3.11 + + ``Py_UNICODE`` is deprecated since Python 3.3. Please migrate to + ``PyObject_CallFunction(PyExc_UnicodeTranslateError, "Onns", ...)``. + .. c:function:: PyObject* PyUnicodeDecodeError_GetEncoding(PyObject *exc) PyObject* PyUnicodeEncodeError_GetEncoding(PyObject *exc) diff --git a/Include/cpython/pyerrors.h b/Include/cpython/pyerrors.h index e3098b3925bd7..418f48a7e2bae 100644 --- a/Include/cpython/pyerrors.h +++ b/Include/cpython/pyerrors.h @@ -148,7 +148,10 @@ PyAPI_FUNC(PyObject *) PyErr_ProgramTextObject( PyObject *filename, int lineno); -/* Create a UnicodeEncodeError object */ +/* Create a UnicodeEncodeError object. + * + * TODO: This API will be removed in Python 3.11. + */ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject *) PyUnicodeEncodeError_Create( const char *encoding, /* UTF-8 encoded string */ const Py_UNICODE *object, @@ -158,7 +161,10 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject *) PyUnicodeEncodeError_Create( const char *reason /* UTF-8 encoded string */ ); -/* Create a UnicodeTranslateError object */ +/* Create a UnicodeTranslateError object. + * + * TODO: This API will be removed in Python 3.11. + */ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject *) PyUnicodeTranslateError_Create( const Py_UNICODE *object, Py_ssize_t length, From webhook-mailer at python.org Fri Aug 7 11:18:42 2020 From: webhook-mailer at python.org (Hai Shi) Date: Fri, 07 Aug 2020 15:18:42 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21764) Message-ID: https://github.com/python/cpython/commit/598a951844122678de2596dbc1e0e09e2be65fd2 commit: 598a951844122678de2596dbc1e0e09e2be65fd2 branch: master author: Hai Shi committer: GitHub date: 2020-08-07T17:18:38+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21764) files: M Lib/ctypes/test/test_loading.py M Lib/sqlite3/test/hooks.py M Lib/test/_test_multiprocessing.py M Lib/test/test___all__.py M Lib/test/test_asyncio/test_proactor_events.py M Lib/test/test_ntpath.py M Lib/test/test_ossaudiodev.py M Lib/test/test_platform.py M Lib/test/test_posixpath.py M Lib/test/test_py_compile.py M Lib/test/test_source_encoding.py M Lib/test/test_startfile.py M Lib/test/test_sundry.py M Lib/test/test_winconsoleio.py M Lib/test/test_zoneinfo/_support.py diff --git a/Lib/ctypes/test/test_loading.py b/Lib/ctypes/test/test_loading.py index ba655bceb8b21..38b45f95fefae 100644 --- a/Lib/ctypes/test/test_loading.py +++ b/Lib/ctypes/test/test_loading.py @@ -5,6 +5,7 @@ import sys import unittest import test.support +from test.support import import_helper from ctypes.util import find_library libc_name = None @@ -117,7 +118,7 @@ def test_1703286_B(self): @unittest.skipUnless(os.name == "nt", 'test specific to Windows') def test_load_dll_with_flags(self): - _sqlite3 = test.support.import_module("_sqlite3") + _sqlite3 = import_helper.import_module("_sqlite3") src = _sqlite3.__file__ if src.lower().endswith("_d.pyd"): ext = "_d.dll" diff --git a/Lib/sqlite3/test/hooks.py b/Lib/sqlite3/test/hooks.py index d74e74bf27227..b08adf1d8097b 100644 --- a/Lib/sqlite3/test/hooks.py +++ b/Lib/sqlite3/test/hooks.py @@ -24,7 +24,8 @@ import unittest import sqlite3 as sqlite -from test.support import TESTFN, unlink +from test.support.os_helper import TESTFN, unlink + class CollationTests(unittest.TestCase): def CheckCreateCollationNotString(self): diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index bde102ae2e051..58663c02002e9 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -28,8 +28,10 @@ from test import support from test.support import hashlib_helper from test.support import import_helper +from test.support import os_helper from test.support import socket_helper from test.support import threading_helper +from test.support import warnings_helper # Skip tests if _multiprocessing wasn't built. @@ -615,7 +617,7 @@ def test_lose_target_ref(self): @classmethod def _test_child_fd_inflation(self, evt, q): - q.put(test.support.fd_count()) + q.put(os_helper.fd_count()) evt.wait() def test_child_fd_inflation(self): @@ -819,8 +821,8 @@ def test_stderr_flush(self): if self.TYPE == "threads": self.skipTest('test not appropriate for {}'.format(self.TYPE)) - testfn = test.support.TESTFN - self.addCleanup(test.support.unlink, testfn) + testfn = os_helper.TESTFN + self.addCleanup(os_helper.unlink, testfn) proc = self.Process(target=self._test_stderr_flush, args=(testfn,)) proc.start() proc.join() @@ -849,8 +851,8 @@ def test_sys_exit(self): if self.TYPE == 'threads': self.skipTest('test not appropriate for {}'.format(self.TYPE)) - testfn = test.support.TESTFN - self.addCleanup(test.support.unlink, testfn) + testfn = os_helper.TESTFN + self.addCleanup(os_helper.unlink, testfn) for reason in ( [1, 2, 3], @@ -1114,7 +1116,7 @@ def test_task_done(self): close_queue(queue) def test_no_import_lock_contention(self): - with test.support.temp_cwd(): + with os_helper.temp_cwd(): module_name = 'imported_by_an_imported_module' with open(module_name + '.py', 'w') as f: f.write("""if 1: @@ -1127,7 +1129,7 @@ def test_no_import_lock_contention(self): del q """) - with test.support.DirsOnSysPath(os.getcwd()): + with import_helper.DirsOnSysPath(os.getcwd()): try: __import__(module_name) except pyqueue.Empty: @@ -2688,8 +2690,8 @@ def test_resource_warning(self): # force state to RUN to emit ResourceWarning in __del__() pool._state = multiprocessing.pool.RUN - with support.check_warnings(('unclosed running multiprocessing pool', - ResourceWarning)): + with warnings_helper.check_warnings( + ('unclosed running multiprocessing pool', ResourceWarning)): pool = None support.gc_collect() @@ -3199,14 +3201,14 @@ def test_fd_transfer(self): p = self.Process(target=self._writefd, args=(child_conn, b"foo")) p.daemon = True p.start() - self.addCleanup(test.support.unlink, test.support.TESTFN) - with open(test.support.TESTFN, "wb") as f: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + with open(os_helper.TESTFN, "wb") as f: fd = f.fileno() if msvcrt: fd = msvcrt.get_osfhandle(fd) reduction.send_handle(conn, fd, p.pid) p.join() - with open(test.support.TESTFN, "rb") as f: + with open(os_helper.TESTFN, "rb") as f: self.assertEqual(f.read(), b"foo") @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction") @@ -3225,8 +3227,8 @@ def test_large_fd_transfer(self): p = self.Process(target=self._writefd, args=(child_conn, b"bar", True)) p.daemon = True p.start() - self.addCleanup(test.support.unlink, test.support.TESTFN) - with open(test.support.TESTFN, "wb") as f: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + with open(os_helper.TESTFN, "wb") as f: fd = f.fileno() for newfd in range(256, MAXFD): if not self._is_fd_assigned(newfd): @@ -3239,7 +3241,7 @@ def test_large_fd_transfer(self): finally: os.close(newfd) p.join() - with open(test.support.TESTFN, "rb") as f: + with open(os_helper.TESTFN, "rb") as f: self.assertEqual(f.read(), b"bar") @classmethod diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py index 0ba243ee4e74e..0a03dd20065fe 100644 --- a/Lib/test/test___all__.py +++ b/Lib/test/test___all__.py @@ -1,5 +1,6 @@ import unittest from test import support +from test.support import warnings_helper import os import sys @@ -15,7 +16,7 @@ class AllTest(unittest.TestCase): def check_all(self, modname): names = {} - with support.check_warnings( + with warnings_helper.check_warnings( (".* (module|package)", DeprecationWarning), (".* (module|package)", PendingDeprecationWarning), ("", ResourceWarning), @@ -31,7 +32,7 @@ def check_all(self, modname): raise NoAll(modname) names = {} with self.subTest(module=modname): - with support.check_warnings( + with warnings_helper.check_warnings( ("", DeprecationWarning), ("", ResourceWarning), quiet=True): diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py index 50ba4c19d425c..d0ab38743ecd1 100644 --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -12,7 +12,7 @@ from asyncio.proactor_events import _ProactorWritePipeTransport from asyncio.proactor_events import _ProactorDuplexPipeTransport from asyncio.proactor_events import _ProactorDatagramTransport -from test import support +from test.support import os_helper from test.support import socket_helper from test.test_asyncio import utils as test_utils @@ -935,20 +935,20 @@ async def wait_closed(self): @classmethod def setUpClass(cls): - with open(support.TESTFN, 'wb') as fp: + with open(os_helper.TESTFN, 'wb') as fp: fp.write(cls.DATA) super().setUpClass() @classmethod def tearDownClass(cls): - support.unlink(support.TESTFN) + os_helper.unlink(os_helper.TESTFN) super().tearDownClass() def setUp(self): self.loop = asyncio.ProactorEventLoop() self.set_event_loop(self.loop) self.addCleanup(self.loop.close) - self.file = open(support.TESTFN, 'rb') + self.file = open(os_helper.TESTFN, 'rb') self.addCleanup(self.file.close) super().setUp() diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py index 0d84ff8bce2c0..a8f764f48ca00 100644 --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -6,7 +6,7 @@ from test.support import os_helper from test.support import TestFailed from test.support.os_helper import FakePath -from test import support, test_genericpath +from test import test_genericpath from tempfile import TemporaryFile @@ -287,7 +287,7 @@ def test_realpath_broken_symlinks(self): os.mkdir(ABSTFN) self.addCleanup(os_helper.rmtree, ABSTFN) - with support.change_cwd(ABSTFN): + with os_helper.change_cwd(ABSTFN): os.mkdir("subdir") os.chdir("subdir") os.symlink(".", "recursive") @@ -443,11 +443,11 @@ def test_realpath_cwd(self): self.assertPathEqual(test_file_long, ntpath.realpath(test_file_short)) - with support.change_cwd(test_dir_long): + with os_helper.change_cwd(test_dir_long): self.assertPathEqual(test_file_long, ntpath.realpath("file.txt")) - with support.change_cwd(test_dir_long.lower()): + with os_helper.change_cwd(test_dir_long.lower()): self.assertPathEqual(test_file_long, ntpath.realpath("file.txt")) - with support.change_cwd(test_dir_short): + with os_helper.change_cwd(test_dir_short): self.assertPathEqual(test_file_long, ntpath.realpath("file.txt")) def test_expandvars(self): @@ -673,7 +673,7 @@ def test_ismount(self): # locations below cannot then refer to mount points # drive, path = ntpath.splitdrive(sys.executable) - with support.change_cwd(ntpath.dirname(sys.executable)): + with os_helper.change_cwd(ntpath.dirname(sys.executable)): self.assertFalse(ntpath.ismount(drive.lower())) self.assertFalse(ntpath.ismount(drive.upper())) diff --git a/Lib/test/test_ossaudiodev.py b/Lib/test/test_ossaudiodev.py index 624fbf21ba7d8..d3766e580b984 100644 --- a/Lib/test/test_ossaudiodev.py +++ b/Lib/test/test_ossaudiodev.py @@ -1,9 +1,10 @@ from test import support +from test.support import import_helper support.requires('audio') from test.support import findfile -ossaudiodev = support.import_module('ossaudiodev') +ossaudiodev = import_helper.import_module('ossaudiodev') import errno import sys diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py index 5ad306e0ed579..b5d21e54610e3 100644 --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -196,7 +196,7 @@ def test_uname_win32_ARCHITEW6432(self): # using it, per # http://blogs.msdn.com/david.wang/archive/2006/03/26/HOWTO-Detect-Process-Bitness.aspx try: - with support.EnvironmentVarGuard() as environ: + with os_helper.EnvironmentVarGuard() as environ: if 'PROCESSOR_ARCHITEW6432' in environ: del environ['PROCESSOR_ARCHITEW6432'] environ['PROCESSOR_ARCHITECTURE'] = 'foo' diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py index 18819a5dc1cfe..f37e82505796d 100644 --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -2,7 +2,9 @@ import posixpath import unittest from posixpath import realpath, abspath, dirname, basename -from test import support, test_genericpath +from test import test_genericpath +from test.support import import_helper +from test.support import os_helper from test.support import FakePath from unittest import mock @@ -15,7 +17,7 @@ # An absolute path to a temporary filename for testing. We can't rely on TESTFN # being an absolute path, so we need this. -ABSTFN = abspath(support.TESTFN) +ABSTFN = abspath(os_helper.TESTFN) def skip_if_ABSTFN_contains_backslash(test): """ @@ -40,8 +42,8 @@ def setUp(self): def tearDown(self): for suffix in ["", "1", "2"]: - support.unlink(support.TESTFN + suffix) - safe_rmdir(support.TESTFN + suffix) + os_helper.unlink(os_helper.TESTFN + suffix) + safe_rmdir(os_helper.TESTFN + suffix) def test_join(self): self.assertEqual(posixpath.join("/foo", "bar", "/bar", "baz"), @@ -152,25 +154,25 @@ def test_dirname(self): self.assertEqual(posixpath.dirname(b"//foo//bar"), b"//foo") def test_islink(self): - self.assertIs(posixpath.islink(support.TESTFN + "1"), False) - self.assertIs(posixpath.lexists(support.TESTFN + "2"), False) + self.assertIs(posixpath.islink(os_helper.TESTFN + "1"), False) + self.assertIs(posixpath.lexists(os_helper.TESTFN + "2"), False) - with open(support.TESTFN + "1", "wb") as f: + with open(os_helper.TESTFN + "1", "wb") as f: f.write(b"foo") - self.assertIs(posixpath.islink(support.TESTFN + "1"), False) + self.assertIs(posixpath.islink(os_helper.TESTFN + "1"), False) - if support.can_symlink(): - os.symlink(support.TESTFN + "1", support.TESTFN + "2") - self.assertIs(posixpath.islink(support.TESTFN + "2"), True) - os.remove(support.TESTFN + "1") - self.assertIs(posixpath.islink(support.TESTFN + "2"), True) - self.assertIs(posixpath.exists(support.TESTFN + "2"), False) - self.assertIs(posixpath.lexists(support.TESTFN + "2"), True) + if os_helper.can_symlink(): + os.symlink(os_helper.TESTFN + "1", os_helper.TESTFN + "2") + self.assertIs(posixpath.islink(os_helper.TESTFN + "2"), True) + os.remove(os_helper.TESTFN + "1") + self.assertIs(posixpath.islink(os_helper.TESTFN + "2"), True) + self.assertIs(posixpath.exists(os_helper.TESTFN + "2"), False) + self.assertIs(posixpath.lexists(os_helper.TESTFN + "2"), True) - self.assertIs(posixpath.islink(support.TESTFN + "\udfff"), False) - self.assertIs(posixpath.islink(os.fsencode(support.TESTFN) + b"\xff"), False) - self.assertIs(posixpath.islink(support.TESTFN + "\x00"), False) - self.assertIs(posixpath.islink(os.fsencode(support.TESTFN) + b"\x00"), False) + self.assertIs(posixpath.islink(os_helper.TESTFN + "\udfff"), False) + self.assertIs(posixpath.islink(os.fsencode(os_helper.TESTFN) + b"\xff"), False) + self.assertIs(posixpath.islink(os_helper.TESTFN + "\x00"), False) + self.assertIs(posixpath.islink(os.fsencode(os_helper.TESTFN) + b"\x00"), False) def test_ismount(self): self.assertIs(posixpath.ismount("/"), True) @@ -190,7 +192,7 @@ def test_ismount_non_existent(self): self.assertIs(posixpath.ismount('/\x00'), False) self.assertIs(posixpath.ismount(b'/\x00'), False) - @unittest.skipUnless(support.can_symlink(), + @unittest.skipUnless(os_helper.can_symlink(), "Test requires symlink support") def test_ismount_symlinks(self): # Symlinks are never mountpoints. @@ -245,7 +247,7 @@ def test_expanduser(self): self.assertEqual(posixpath.expanduser(b"foo"), b"foo") def test_expanduser_home_envvar(self): - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env['HOME'] = '/home/victor' self.assertEqual(posixpath.expanduser("~"), "/home/victor") @@ -261,7 +263,7 @@ def test_expanduser_home_envvar(self): self.assertEqual(posixpath.expanduser("~/foo"), "/foo") def test_expanduser_pwd(self): - pwd = support.import_module('pwd') + pwd = import_helper.import_module('pwd') self.assertIsInstance(posixpath.expanduser("~/"), str) self.assertIsInstance(posixpath.expanduser(b"~/"), bytes) @@ -281,7 +283,7 @@ def test_expanduser_pwd(self): self.assertIsInstance(posixpath.expanduser(b"~root/"), bytes) self.assertIsInstance(posixpath.expanduser(b"~foo/"), bytes) - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: # expanduser should fall back to using the password database del env['HOME'] @@ -348,7 +350,7 @@ def test_realpath_basic(self): os.symlink(ABSTFN+"1", ABSTFN) self.assertEqual(realpath(ABSTFN), ABSTFN+"1") finally: - support.unlink(ABSTFN) + os_helper.unlink(ABSTFN) @unittest.skipUnless(hasattr(os, "symlink"), "Missing symlink implementation") @@ -358,7 +360,7 @@ def test_realpath_relative(self): os.symlink(posixpath.relpath(ABSTFN+"1"), ABSTFN) self.assertEqual(realpath(ABSTFN), ABSTFN+"1") finally: - support.unlink(ABSTFN) + os_helper.unlink(ABSTFN) @unittest.skipUnless(hasattr(os, "symlink"), "Missing symlink implementation") @@ -392,15 +394,15 @@ def test_realpath_symlink_loops(self): self.assertEqual(realpath(ABSTFN+"c"), ABSTFN+"c") # Test using relative path as well. - with support.change_cwd(dirname(ABSTFN)): + with os_helper.change_cwd(dirname(ABSTFN)): self.assertEqual(realpath(basename(ABSTFN)), ABSTFN) finally: - support.unlink(ABSTFN) - support.unlink(ABSTFN+"1") - support.unlink(ABSTFN+"2") - support.unlink(ABSTFN+"y") - support.unlink(ABSTFN+"c") - support.unlink(ABSTFN+"a") + os_helper.unlink(ABSTFN) + os_helper.unlink(ABSTFN+"1") + os_helper.unlink(ABSTFN+"2") + os_helper.unlink(ABSTFN+"y") + os_helper.unlink(ABSTFN+"c") + os_helper.unlink(ABSTFN+"a") @unittest.skipUnless(hasattr(os, "symlink"), "Missing symlink implementation") @@ -413,8 +415,8 @@ def test_realpath_repeated_indirect_symlinks(self): os.symlink('self/self/self', ABSTFN + '/link') self.assertEqual(realpath(ABSTFN + '/link'), ABSTFN) finally: - support.unlink(ABSTFN + '/self') - support.unlink(ABSTFN + '/link') + os_helper.unlink(ABSTFN + '/self') + os_helper.unlink(ABSTFN + '/link') safe_rmdir(ABSTFN) @unittest.skipUnless(hasattr(os, "symlink"), @@ -430,11 +432,11 @@ def test_realpath_deep_recursion(self): self.assertEqual(realpath(ABSTFN + '/%d' % depth), ABSTFN) # Test using relative path as well. - with support.change_cwd(ABSTFN): + with os_helper.change_cwd(ABSTFN): self.assertEqual(realpath('%d' % depth), ABSTFN) finally: for i in range(depth + 1): - support.unlink(ABSTFN + '/%d' % i) + os_helper.unlink(ABSTFN + '/%d' % i) safe_rmdir(ABSTFN) @unittest.skipUnless(hasattr(os, "symlink"), @@ -450,10 +452,10 @@ def test_realpath_resolve_parents(self): os.mkdir(ABSTFN + "/y") os.symlink(ABSTFN + "/y", ABSTFN + "/k") - with support.change_cwd(ABSTFN + "/k"): + with os_helper.change_cwd(ABSTFN + "/k"): self.assertEqual(realpath("a"), ABSTFN + "/y/a") finally: - support.unlink(ABSTFN + "/k") + os_helper.unlink(ABSTFN + "/k") safe_rmdir(ABSTFN + "/y") safe_rmdir(ABSTFN) @@ -477,11 +479,11 @@ def test_realpath_resolve_before_normalizing(self): # Absolute path. self.assertEqual(realpath(ABSTFN + "/link-y/.."), ABSTFN + "/k") # Relative path. - with support.change_cwd(dirname(ABSTFN)): + with os_helper.change_cwd(dirname(ABSTFN)): self.assertEqual(realpath(basename(ABSTFN) + "/link-y/.."), ABSTFN + "/k") finally: - support.unlink(ABSTFN + "/link-y") + os_helper.unlink(ABSTFN + "/link-y") safe_rmdir(ABSTFN + "/k/y") safe_rmdir(ABSTFN + "/k") safe_rmdir(ABSTFN) @@ -497,12 +499,12 @@ def test_realpath_resolve_first(self): os.mkdir(ABSTFN) os.mkdir(ABSTFN + "/k") os.symlink(ABSTFN, ABSTFN + "link") - with support.change_cwd(dirname(ABSTFN)): + with os_helper.change_cwd(dirname(ABSTFN)): base = basename(ABSTFN) self.assertEqual(realpath(base + "link"), ABSTFN) self.assertEqual(realpath(base + "link/k"), ABSTFN + "/k") finally: - support.unlink(ABSTFN + "link") + os_helper.unlink(ABSTFN + "link") safe_rmdir(ABSTFN + "/k") safe_rmdir(ABSTFN) @@ -627,9 +629,9 @@ class PathLikeTests(unittest.TestCase): path = posixpath def setUp(self): - self.file_name = support.TESTFN - self.file_path = FakePath(support.TESTFN) - self.addCleanup(support.unlink, self.file_name) + self.file_name = os_helper.TESTFN + self.file_path = FakePath(os_helper.TESTFN) + self.addCleanup(os_helper.unlink, self.file_name) with open(self.file_name, 'xb', 0) as file: file.write(b"test_posixpath.PathLikeTests") diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py index 009645689f423..b58f28a4bc888 100644 --- a/Lib/test/test_py_compile.py +++ b/Lib/test/test_py_compile.py @@ -228,7 +228,7 @@ def setUp(self): file.write('x = 123\n') def tearDown(self): - support.rmtree(self.directory) + os_helper.rmtree(self.directory) def pycompilecmd(self, *args, **kwargs): # assert_python_* helpers don't return proc object. We'll just use diff --git a/Lib/test/test_source_encoding.py b/Lib/test/test_source_encoding.py index 5ca43461d9940..b410c03221bf3 100644 --- a/Lib/test/test_source_encoding.py +++ b/Lib/test/test_source_encoding.py @@ -1,7 +1,9 @@ # -*- coding: koi8-r -*- import unittest -from test.support import TESTFN, unlink, unload, rmtree, script_helper, captured_stdout +from test.support import script_helper, captured_stdout +from test.support.os_helper import TESTFN, unlink, rmtree +from test.support.import_helper import unload import importlib import os import sys diff --git a/Lib/test/test_startfile.py b/Lib/test/test_startfile.py index 1a26a8025e624..589ffa25244da 100644 --- a/Lib/test/test_startfile.py +++ b/Lib/test/test_startfile.py @@ -9,6 +9,7 @@ import unittest from test import support +from test.support import os_helper import os import platform import sys @@ -27,7 +28,7 @@ def test_empty(self): # we're not about to delete. If we're running under -j, that # means the test harness provided directory isn't a safe option. # See http://bugs.python.org/issue15526 for more details - with support.change_cwd(path.dirname(sys.executable)): + with os_helper.change_cwd(path.dirname(sys.executable)): empty = path.join(path.dirname(__file__), "empty.vbs") startfile(empty) startfile(empty, "open") diff --git a/Lib/test/test_sundry.py b/Lib/test/test_sundry.py index 2accad1aeebd4..04e572c00a196 100644 --- a/Lib/test/test_sundry.py +++ b/Lib/test/test_sundry.py @@ -3,15 +3,17 @@ import platform import sys from test import support +from test.support import import_helper +from test.support import warnings_helper import unittest class TestUntestedModules(unittest.TestCase): def test_untested_modules_can_be_imported(self): untested = ('encodings', 'formatter') - with support.check_warnings(quiet=True): + with warnings_helper.check_warnings(quiet=True): for name in untested: try: - support.import_module('test.test_{}'.format(name)) + import_helper.import_module('test.test_{}'.format(name)) except unittest.SkipTest: importlib.import_module(name) else: diff --git a/Lib/test/test_winconsoleio.py b/Lib/test/test_winconsoleio.py index a44f7bbd27b70..1807e47c66c38 100644 --- a/Lib/test/test_winconsoleio.py +++ b/Lib/test/test_winconsoleio.py @@ -6,7 +6,7 @@ import sys import tempfile import unittest -from test import support +from test.support import os_helper if sys.platform != 'win32': raise unittest.SkipTest("test only relevant on win32") @@ -109,7 +109,7 @@ def test_conin_conout_names(self): def test_conout_path(self): temp_path = tempfile.mkdtemp() - self.addCleanup(support.rmtree, temp_path) + self.addCleanup(os_helper.rmtree, temp_path) conout_path = os.path.join(temp_path, 'CONOUT$') diff --git a/Lib/test/test_zoneinfo/_support.py b/Lib/test/test_zoneinfo/_support.py index 0fe162c258368..5a76c163fb75b 100644 --- a/Lib/test/test_zoneinfo/_support.py +++ b/Lib/test/test_zoneinfo/_support.py @@ -3,7 +3,7 @@ import sys import threading import unittest -from test.support import import_fresh_module +from test.support.import_helper import import_fresh_module OS_ENV_LOCK = threading.Lock() TZPATH_LOCK = threading.Lock() From webhook-mailer at python.org Fri Aug 7 11:57:02 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Aug 2020 15:57:02 -0000 Subject: [Python-checkins] bpo-41477: Make ctypes optional in test_genericalias (GH-21766) Message-ID: https://github.com/python/cpython/commit/f44693eaed3b9d91a6e415d48224fd4750b59366 commit: f44693eaed3b9d91a6e415d48224fd4750b59366 branch: master author: Victor Stinner committer: GitHub date: 2020-08-07T17:56:42+02:00 summary: bpo-41477: Make ctypes optional in test_genericalias (GH-21766) files: A Misc/NEWS.d/next/Tests/2020-08-07-17-28-49.bpo-41477.GrFexU.rst M Lib/test/test_genericalias.py diff --git a/Lib/test/test_genericalias.py b/Lib/test/test_genericalias.py index 4f3798e8f87d8..1f24469471428 100644 --- a/Lib/test/test_genericalias.py +++ b/Lib/test/test_genericalias.py @@ -13,7 +13,10 @@ from dataclasses import Field from functools import partial, partialmethod, cached_property from mailbox import Mailbox, _PartialFile -from ctypes import Array, LibraryLoader +try: + import ctypes +except ImportError: + ctypes = None from difflib import SequenceMatcher from filecmp import dircmp from fileinput import FileInput @@ -46,43 +49,44 @@ class BaseTest(unittest.TestCase): """Test basics.""" def test_subscriptable(self): - for t in (type, tuple, list, dict, set, frozenset, enumerate, - defaultdict, deque, - SequenceMatcher, - dircmp, - FileInput, - OrderedDict, Counter, UserDict, UserList, - Pattern, Match, - partial, partialmethod, cached_property, - AbstractContextManager, AbstractAsyncContextManager, - Awaitable, Coroutine, - AsyncIterable, AsyncIterator, - AsyncGenerator, Generator, - Iterable, Iterator, - Reversible, - Container, Collection, - Callable, - Mailbox, _PartialFile, - ContextVar, Token, - Field, - Set, MutableSet, - Mapping, MutableMapping, MappingView, - KeysView, ItemsView, ValuesView, - Sequence, MutableSequence, - MappingProxyType, AsyncGeneratorType, - DirEntry, - chain, - TemporaryDirectory, SpooledTemporaryFile, - Queue, SimpleQueue, - _AssertRaisesContext, - Array, LibraryLoader, - SplitResult, ParseResult, - ValueProxy, ApplyResult, - WeakSet, ReferenceType, ref, - ShareableList, SimpleQueue, - Future, _WorkItem, - Morsel, - ): + types = [type, tuple, list, dict, set, frozenset, enumerate, + defaultdict, deque, + SequenceMatcher, + dircmp, + FileInput, + OrderedDict, Counter, UserDict, UserList, + Pattern, Match, + partial, partialmethod, cached_property, + AbstractContextManager, AbstractAsyncContextManager, + Awaitable, Coroutine, + AsyncIterable, AsyncIterator, + AsyncGenerator, Generator, + Iterable, Iterator, + Reversible, + Container, Collection, + Callable, + Mailbox, _PartialFile, + ContextVar, Token, + Field, + Set, MutableSet, + Mapping, MutableMapping, MappingView, + KeysView, ItemsView, ValuesView, + Sequence, MutableSequence, + MappingProxyType, AsyncGeneratorType, + DirEntry, + chain, + TemporaryDirectory, SpooledTemporaryFile, + Queue, SimpleQueue, + _AssertRaisesContext, + SplitResult, ParseResult, + ValueProxy, ApplyResult, + WeakSet, ReferenceType, ref, + ShareableList, SimpleQueue, + Future, _WorkItem, + Morsel] + if ctypes is not None: + types.extend((ctypes.Array, ctypes.LibraryLoader)) + for t in types: if t is None: continue tname = t.__name__ diff --git a/Misc/NEWS.d/next/Tests/2020-08-07-17-28-49.bpo-41477.GrFexU.rst b/Misc/NEWS.d/next/Tests/2020-08-07-17-28-49.bpo-41477.GrFexU.rst new file mode 100644 index 0000000000000..bf0f54abecd85 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2020-08-07-17-28-49.bpo-41477.GrFexU.rst @@ -0,0 +1 @@ +Make ctypes optional in test_genericalias. From webhook-mailer at python.org Fri Aug 7 11:58:00 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 07 Aug 2020 15:58:00 -0000 Subject: [Python-checkins] bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768) Message-ID: https://github.com/python/cpython/commit/e27a51c11e10d5df79b3e48dc3e7bfedfad5a794 commit: e27a51c11e10d5df79b3e48dc3e7bfedfad5a794 branch: master author: Victor Stinner committer: GitHub date: 2020-08-07T17:57:56+02:00 summary: bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768) gdb 9.2 on Fedora Rawhide is not reliable, see: * https://bugs.python.org/issue41473 * https://bugzilla.redhat.com/show_bug.cgi?id=1866884 files: M Lib/test/test_gdb.py diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index 22c75bae98721..44cb9a0f07b75 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -51,6 +51,11 @@ def get_gdb_version(): "embedding. Saw %s.%s:\n%s" % (gdb_major_version, gdb_minor_version, gdb_version)) +if (gdb_major_version, gdb_minor_version) >= (9, 2): + # gdb 9.2 on Fedora Rawhide is not reliable, see: + # * https://bugs.python.org/issue41473 + # * https://bugzilla.redhat.com/show_bug.cgi?id=1866884 + raise unittest.SkipTest("https://bugzilla.redhat.com/show_bug.cgi?id=1866884") if not sysconfig.is_python_build(): raise unittest.SkipTest("test_gdb only works on source builds at the moment.") From webhook-mailer at python.org Fri Aug 7 12:15:41 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 07 Aug 2020 16:15:41 -0000 Subject: [Python-checkins] bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768) Message-ID: https://github.com/python/cpython/commit/87bc22051fcfcf181160d06a57ac2b16ee071f8f commit: 87bc22051fcfcf181160d06a57ac2b16ee071f8f branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-07T09:15:37-07:00 summary: bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768) gdb 9.2 on Fedora Rawhide is not reliable, see: * https://bugs.python.org/issue41473 * https://bugzilla.redhat.com/show_bug.cgi?id=1866884 (cherry picked from commit e27a51c11e10d5df79b3e48dc3e7bfedfad5a794) Co-authored-by: Victor Stinner files: M Lib/test/test_gdb.py diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index d90ca5a51ae1b..01a48af285422 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -51,6 +51,11 @@ def get_gdb_version(): "embedding. Saw %s.%s:\n%s" % (gdb_major_version, gdb_minor_version, gdb_version)) +if (gdb_major_version, gdb_minor_version) >= (9, 2): + # gdb 9.2 on Fedora Rawhide is not reliable, see: + # * https://bugs.python.org/issue41473 + # * https://bugzilla.redhat.com/show_bug.cgi?id=1866884 + raise unittest.SkipTest("https://bugzilla.redhat.com/show_bug.cgi?id=1866884") if not sysconfig.is_python_build(): raise unittest.SkipTest("test_gdb only works on source builds at the moment.") From webhook-mailer at python.org Fri Aug 7 17:55:43 2020 From: webhook-mailer at python.org (Hai Shi) Date: Fri, 07 Aug 2020 21:55:43 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21772) Message-ID: https://github.com/python/cpython/commit/fcce8c649a14f7a81fae82f9f203bb5b5ee0c205 commit: fcce8c649a14f7a81fae82f9f203bb5b5ee0c205 branch: master author: Hai Shi committer: GitHub date: 2020-08-07T23:55:35+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21772) files: M Lib/sqlite3/test/dbapi.py M Lib/test/libregrtest/refleak.py M Lib/test/libregrtest/win_utils.py M Lib/test/test_audit.py M Lib/test/test_bytes.py M Lib/test/test_dbm_ndbm.py M Lib/test/test_descr.py M Lib/test/test_xml_etree.py diff --git a/Lib/sqlite3/test/dbapi.py b/Lib/sqlite3/test/dbapi.py index be11337154bdd..119da12170331 100644 --- a/Lib/sqlite3/test/dbapi.py +++ b/Lib/sqlite3/test/dbapi.py @@ -25,7 +25,7 @@ import unittest import sqlite3 as sqlite -from test.support import TESTFN, unlink +from test.support.os_helper import TESTFN, unlink class ModuleTests(unittest.TestCase): diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index 8d221232eb6ce..77298d318898d 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -4,6 +4,8 @@ import warnings from inspect import isabstract from test import support +from test.support import os_helper + try: from _abc import _get_dump except ImportError: @@ -61,7 +63,7 @@ def get_pooled_int(value): return int_pool.setdefault(value, value) nwarmup, ntracked, fname = ns.huntrleaks - fname = os.path.join(support.SAVEDCWD, fname) + fname = os.path.join(os_helper.SAVEDCWD, fname) repcount = nwarmup + ntracked # Pre-allocate to ensure that the loop doesn't allocate anything new @@ -71,7 +73,7 @@ def get_pooled_int(value): fd_deltas = [0] * repcount getallocatedblocks = sys.getallocatedblocks gettotalrefcount = sys.gettotalrefcount - fd_count = support.fd_count + fd_count = os_helper.fd_count # initialize variables to make pyflakes quiet rc_before = alloc_before = fd_before = 0 diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py index 028c01106dee0..a1cc2201147db 100644 --- a/Lib/test/libregrtest/win_utils.py +++ b/Lib/test/libregrtest/win_utils.py @@ -5,7 +5,7 @@ import subprocess import uuid import winreg -from test import support +from test.support import os_helper from test.libregrtest.utils import print_warning @@ -69,7 +69,9 @@ def start(self): # Spawn off the load monitor counter_name = self._get_counter_name() command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)] - self._popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD) + self._popen = subprocess.Popen(' '.join(command), + stdout=command_stdout, + cwd=os_helper.SAVEDCWD) # Close our copy of the write end of the pipe os.close(command_stdout) diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index f79edbc4bd0d9..4f8d06a3ebcbe 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -5,6 +5,9 @@ import sys import unittest from test import support +from test.support import import_helper +from test.support import os_helper + if not hasattr(sys, "addaudithook") or not hasattr(sys, "audit"): raise unittest.SkipTest("test only relevant when sys.audit is available") @@ -52,7 +55,7 @@ def test_block_add_hook_baseexception(self): self.do_test("test_block_add_hook_baseexception") def test_pickle(self): - support.import_module("pickle") + import_helper.import_module("pickle") self.do_test("test_pickle") @@ -60,7 +63,7 @@ def test_monkeypatch(self): self.do_test("test_monkeypatch") def test_open(self): - self.do_test("test_open", support.TESTFN) + self.do_test("test_open", os_helper.TESTFN) def test_cantrace(self): self.do_test("test_cantrace") @@ -89,7 +92,7 @@ def test_unraisablehook(self): ) def test_winreg(self): - support.import_module("winreg") + import_helper.import_module("winreg") returncode, events, stderr = self.run_python("test_winreg") if returncode: self.fail(stderr) @@ -103,7 +106,7 @@ def test_winreg(self): self.assertSequenceEqual(["winreg.PyHKEY.Detach", " ", expected], events[4]) def test_socket(self): - support.import_module("socket") + import_helper.import_module("socket") returncode, events, stderr = self.run_python("test_socket") if returncode: self.fail(stderr) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py index 61b4b9162ccc5..e61228d1a266f 100644 --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -17,6 +17,7 @@ import test.support from test.support import import_helper +from test.support import warnings_helper import test.string_tests import test.list_tests from test.support import bigaddrspacetest, MAX_Py_ssize_t @@ -27,7 +28,7 @@ def check_bytes_warnings(func): @functools.wraps(func) def wrapper(*args, **kw): - with test.support.check_warnings(('', BytesWarning)): + with warnings_helper.check_warnings(('', BytesWarning)): return func(*args, **kw) return wrapper else: @@ -1769,7 +1770,7 @@ def test_return_self(self): "BytesWarning is needed for this test: use -bb option") def test_compare(self): def bytes_warning(): - return test.support.check_warnings(('', BytesWarning)) + return warnings_helper.check_warnings(('', BytesWarning)) with bytes_warning(): b'' == '' with bytes_warning(): diff --git a/Lib/test/test_dbm_ndbm.py b/Lib/test/test_dbm_ndbm.py index 278fca2cf2315..e17a1d9eca931 100644 --- a/Lib/test/test_dbm_ndbm.py +++ b/Lib/test/test_dbm_ndbm.py @@ -1,5 +1,6 @@ from test import support from test.support import import_helper +from test.support import os_helper import_helper.import_module("dbm.ndbm") #skip if not supported import os import unittest @@ -9,7 +10,7 @@ class DbmTestCase(unittest.TestCase): def setUp(self): - self.filename = support.TESTFN + self.filename = os_helper.TESTFN self.d = dbm.ndbm.open(self.filename, 'c') self.d.close() @@ -102,10 +103,10 @@ def test_write_readonly_file(self): with self.assertRaises(error): db[b'not exist key'] = b'not exist value' - @unittest.skipUnless(support.TESTFN_NONASCII, + @unittest.skipUnless(os_helper.TESTFN_NONASCII, 'requires OS support of non-ASCII encodings') def test_nonascii_filename(self): - filename = support.TESTFN_NONASCII + filename = os_helper.TESTFN_NONASCII for suffix in ['', '.pag', '.dir', '.db']: self.addCleanup(support.unlink, filename + suffix) with dbm.ndbm.open(filename, 'c') as db: diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 9738fb52a04bf..307416c3300ae 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -2975,12 +2975,12 @@ class sublist(list): ## self.ateof = 1 ## return s ## - ## f = file(name=support.TESTFN, mode='w') + ## f = file(name=os_helper.TESTFN, mode='w') ## lines = ['a\n', 'b\n', 'c\n'] ## try: ## f.writelines(lines) ## f.close() - ## f = CountedInput(support.TESTFN) + ## f = CountedInput(os_helper.TESTFN) ## for (i, expected) in zip(range(1, 5) + [4], lines + 2 * [""]): ## got = f.readline() ## self.assertEqual(expected, got) @@ -2992,7 +2992,7 @@ class sublist(list): ## f.close() ## except: ## pass - ## support.unlink(support.TESTFN) + ## os_helper.unlink(os_helper.TESTFN) def test_keywords(self): # Testing keyword args to basic type constructors ... diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 63f9b92a83de2..c7d446185cfe4 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -111,7 +111,7 @@ def checkwarnings(*filters, quiet=False): def decorator(test): def newtest(*args, **kwargs): - with support.check_warnings(*filters, quiet=quiet): + with warnings_helper.check_warnings(*filters, quiet=quiet): test(*args, **kwargs) functools.update_wrapper(newtest, test) return newtest From webhook-mailer at python.org Fri Aug 7 18:10:41 2020 From: webhook-mailer at python.org (Steve Dower) Date: Fri, 07 Aug 2020 22:10:41 -0000 Subject: [Python-checkins] bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 (GH-21775) Message-ID: https://github.com/python/cpython/commit/135de08128a76f49752ac57c316129500275e828 commit: 135de08128a76f49752ac57c316129500275e828 branch: 3.8 author: Steve Dower committer: GitHub date: 2020-08-07T23:10:32+01:00 summary: bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 (GH-21775) files: A Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl A Lib/ensurepip/_bundled/setuptools-49.2.1-py3-none-any.whl A Misc/NEWS.d/next/Library/2020-08-05-23-16-39.bpo-41490.6z47A_.rst D Lib/ensurepip/_bundled/pip-20.1.1-py2.py3-none-any.whl D Lib/ensurepip/_bundled/setuptools-47.1.0-py3-none-any.whl M .github/workflows/build_msi.yml M Lib/ensurepip/__init__.py M Lib/test/test_importlib/test_resource.py diff --git a/.github/workflows/build_msi.yml b/.github/workflows/build_msi.yml index 769b3d012e940..e35ebe4256522 100644 --- a/.github/workflows/build_msi.yml +++ b/.github/workflows/build_msi.yml @@ -8,6 +8,7 @@ on: - 3.7 paths: - 'Tools/msi/**' + - 'Lib/ensurepip/**' pull_request: branches: - master @@ -15,6 +16,7 @@ on: - 3.7 paths: - 'Tools/msi/**' + - 'Lib/ensurepip/**' jobs: build_win32: diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py index f3152a55d4430..9415fd73b80dd 100644 --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -9,9 +9,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "47.1.0" +_SETUPTOOLS_VERSION = "49.2.1" -_PIP_VERSION = "20.1.1" +_PIP_VERSION = "20.2.1" _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION, "py3"), diff --git a/Lib/ensurepip/_bundled/pip-20.1.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl similarity index 54% rename from Lib/ensurepip/_bundled/pip-20.1.1-py2.py3-none-any.whl rename to Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl index ea1d0f7c8604a..3d0d3f8ae7fac 100644 Binary files a/Lib/ensurepip/_bundled/pip-20.1.1-py2.py3-none-any.whl and b/Lib/ensurepip/_bundled/pip-20.2.1-py2.py3-none-any.whl differ diff --git a/Lib/ensurepip/_bundled/setuptools-47.1.0-py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-49.2.1-py3-none-any.whl similarity index 54% rename from Lib/ensurepip/_bundled/setuptools-47.1.0-py3-none-any.whl rename to Lib/ensurepip/_bundled/setuptools-49.2.1-py3-none-any.whl index f87867ff98254..308e2f2ed5ed9 100644 Binary files a/Lib/ensurepip/_bundled/setuptools-47.1.0-py3-none-any.whl and b/Lib/ensurepip/_bundled/setuptools-49.2.1-py3-none-any.whl differ diff --git a/Lib/test/test_importlib/test_resource.py b/Lib/test/test_importlib/test_resource.py index f88d92d154672..e132c57282b37 100644 --- a/Lib/test/test_importlib/test_resource.py +++ b/Lib/test/test_importlib/test_resource.py @@ -1,10 +1,13 @@ import sys import unittest +import uuid from . import data01 from . import zipdata01, zipdata02 from . import util from importlib import resources, import_module +from pathlib import Path +from test import support class ResourceTests: @@ -162,5 +165,71 @@ def test_namespaces_cannot_have_resources(self): 'test.test_importlib.data03.namespace', 'resource1.txt') +class DeletingZipsTest(unittest.TestCase): + """Having accessed resources in a zip file should not keep an open + reference to the zip. + """ + ZIP_MODULE = zipdata01 + + def setUp(self): + modules = support.modules_setup() + self.addCleanup(support.modules_cleanup, *modules) + + data_path = Path(self.ZIP_MODULE.__file__) + data_dir = data_path.parent + self.source_zip_path = data_dir / 'ziptestdata.zip' + self.zip_path = Path.cwd() / '{}.zip'.format(uuid.uuid4()) + self.zip_path.write_bytes(self.source_zip_path.read_bytes()) + sys.path.append(str(self.zip_path)) + self.data = import_module('ziptestdata') + + def tearDown(self): + try: + sys.path.remove(str(self.zip_path)) + except ValueError: + pass + + try: + del sys.path_importer_cache[str(self.zip_path)] + del sys.modules[self.data.__name__] + except KeyError: + pass + + try: + support.unlink(self.zip_path) + except OSError: + # If the test fails, this will probably fail too + pass + + def test_contents_does_not_keep_open(self): + c = resources.contents('ziptestdata') + self.zip_path.unlink() + + def test_is_resource_does_not_keep_open(self): + c = resources.is_resource('ziptestdata', 'binary.file') + self.zip_path.unlink() + + def test_is_resource_failure_does_not_keep_open(self): + c = resources.is_resource('ziptestdata', 'not-present') + self.zip_path.unlink() + + def test_path_does_not_keep_open(self): + c = resources.path('ziptestdata', 'binary.file') + self.zip_path.unlink() + + def test_entered_path_does_not_keep_open(self): + # This is what certifi does on import to make its bundle + # available for the process duration. + c = resources.path('ziptestdata', 'binary.file').__enter__() + self.zip_path.unlink() + + def test_read_binary_does_not_keep_open(self): + c = resources.read_binary('ziptestdata', 'binary.file') + self.zip_path.unlink() + + def test_read_text_does_not_keep_open(self): + c = resources.read_text('ziptestdata', 'utf-8.file', encoding='utf-8') + self.zip_path.unlink() + if __name__ == '__main__': unittest.main() diff --git a/Misc/NEWS.d/next/Library/2020-08-05-23-16-39.bpo-41490.6z47A_.rst b/Misc/NEWS.d/next/Library/2020-08-05-23-16-39.bpo-41490.6z47A_.rst new file mode 100644 index 0000000000000..e89180db00429 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-05-23-16-39.bpo-41490.6z47A_.rst @@ -0,0 +1 @@ +Update :mod:`ensurepip` to install pip 20.2.1 and setuptools 49.2.1. From webhook-mailer at python.org Fri Aug 7 18:22:10 2020 From: webhook-mailer at python.org (Steve Dower) Date: Fri, 07 Aug 2020 22:22:10 -0000 Subject: [Python-checkins] Update Azure Pipelines build to use Ubuntu 18.04 and move triggers into YAML files (GH-21776) Message-ID: https://github.com/python/cpython/commit/102b4988b1a10d5a61034381aea15521d17c210c commit: 102b4988b1a10d5a61034381aea15521d17c210c branch: master author: Steve Dower committer: GitHub date: 2020-08-07T23:22:02+01:00 summary: Update Azure Pipelines build to use Ubuntu 18.04 and move triggers into YAML files (GH-21776) files: M .azure-pipelines/ci.yml M .azure-pipelines/pr.yml diff --git a/.azure-pipelines/ci.yml b/.azure-pipelines/ci.yml index 50dc50a654934..531ed060fd386 100644 --- a/.azure-pipelines/ci.yml +++ b/.azure-pipelines/ci.yml @@ -1,18 +1,14 @@ variables: - manylinux: false coverage: false -resources: - containers: - - container: manylinux1 - image: pyca/cryptography-manylinux1:x86_64 +trigger: ['master', '3.9', '3.8', '3.7'] jobs: - job: Prebuild displayName: Pre-build checks pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./prebuild-checks.yml @@ -24,7 +20,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['docs.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./docs-steps.yml @@ -56,7 +52,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(build.sourceBranchName)-linux' @@ -69,37 +65,6 @@ jobs: dependencies: apt -- job: ManyLinux1_CI_Tests - displayName: ManyLinux1 CI Tests - dependsOn: Prebuild - condition: | - and( - and( - succeeded(), - eq(variables['manylinux'], 'true') - ), - eq(dependencies.Prebuild.outputs['tests.run'], 'true') - ) - - pool: - vmImage: ubuntu-16.04 - - container: manylinux1 - - variables: - testRunTitle: '$(build.sourceBranchName)-manylinux1' - testRunPlatform: manylinux1 - openssl_version: '' - - steps: - - template: ./posix-steps.yml - parameters: - dependencies: yum - sudo_dependencies: '' - xvfb: false - patchcheck: false - - - job: Ubuntu_Coverage_CI_Tests displayName: Ubuntu CI Tests (coverage) dependsOn: Prebuild @@ -113,7 +78,7 @@ jobs: ) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(Build.SourceBranchName)-linux-coverage' diff --git a/.azure-pipelines/pr.yml b/.azure-pipelines/pr.yml index 228f9db4f8ef2..1ffe0a97a2465 100644 --- a/.azure-pipelines/pr.yml +++ b/.azure-pipelines/pr.yml @@ -1,18 +1,14 @@ variables: - manylinux: false coverage: false -resources: - containers: - - container: manylinux1 - image: pyca/cryptography-manylinux1:x86_64 +pr: ['master', '3.9', '3.8', '3.7'] jobs: - job: Prebuild displayName: Pre-build checks pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./prebuild-checks.yml @@ -24,7 +20,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['docs.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./docs-steps.yml @@ -56,7 +52,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(system.pullRequest.TargetBranch)-linux' @@ -69,37 +65,6 @@ jobs: dependencies: apt -- job: ManyLinux1_PR_Tests - displayName: ManyLinux1 PR Tests - dependsOn: Prebuild - condition: | - and( - and( - succeeded(), - eq(variables['manylinux'], 'true') - ), - eq(dependencies.Prebuild.outputs['tests.run'], 'true') - ) - - pool: - vmImage: ubuntu-16.04 - - container: manylinux1 - - variables: - testRunTitle: '$(system.pullRequest.TargetBranch)-manylinux1' - testRunPlatform: manylinux1 - openssl_version: '' - - steps: - - template: ./posix-steps.yml - parameters: - dependencies: yum - sudo_dependencies: '' - xvfb: false - patchcheck: false - - - job: Ubuntu_Coverage_PR_Tests displayName: Ubuntu PR Tests (coverage) dependsOn: Prebuild @@ -113,7 +78,7 @@ jobs: ) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(Build.SourceBranchName)-linux-coverage' From webhook-mailer at python.org Fri Aug 7 19:01:45 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 07 Aug 2020 23:01:45 -0000 Subject: [Python-checkins] Update Azure Pipelines build to use Ubuntu 18.04 and move triggers into YAML files (GH-21776) Message-ID: https://github.com/python/cpython/commit/398dabb6818992efac3b4df96255054a456907a3 commit: 398dabb6818992efac3b4df96255054a456907a3 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-07T16:01:37-07:00 summary: Update Azure Pipelines build to use Ubuntu 18.04 and move triggers into YAML files (GH-21776) (cherry picked from commit 102b4988b1a10d5a61034381aea15521d17c210c) Co-authored-by: Steve Dower files: M .azure-pipelines/ci.yml M .azure-pipelines/pr.yml diff --git a/.azure-pipelines/ci.yml b/.azure-pipelines/ci.yml index 50dc50a654934..531ed060fd386 100644 --- a/.azure-pipelines/ci.yml +++ b/.azure-pipelines/ci.yml @@ -1,18 +1,14 @@ variables: - manylinux: false coverage: false -resources: - containers: - - container: manylinux1 - image: pyca/cryptography-manylinux1:x86_64 +trigger: ['master', '3.9', '3.8', '3.7'] jobs: - job: Prebuild displayName: Pre-build checks pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./prebuild-checks.yml @@ -24,7 +20,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['docs.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./docs-steps.yml @@ -56,7 +52,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(build.sourceBranchName)-linux' @@ -69,37 +65,6 @@ jobs: dependencies: apt -- job: ManyLinux1_CI_Tests - displayName: ManyLinux1 CI Tests - dependsOn: Prebuild - condition: | - and( - and( - succeeded(), - eq(variables['manylinux'], 'true') - ), - eq(dependencies.Prebuild.outputs['tests.run'], 'true') - ) - - pool: - vmImage: ubuntu-16.04 - - container: manylinux1 - - variables: - testRunTitle: '$(build.sourceBranchName)-manylinux1' - testRunPlatform: manylinux1 - openssl_version: '' - - steps: - - template: ./posix-steps.yml - parameters: - dependencies: yum - sudo_dependencies: '' - xvfb: false - patchcheck: false - - - job: Ubuntu_Coverage_CI_Tests displayName: Ubuntu CI Tests (coverage) dependsOn: Prebuild @@ -113,7 +78,7 @@ jobs: ) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(Build.SourceBranchName)-linux-coverage' diff --git a/.azure-pipelines/pr.yml b/.azure-pipelines/pr.yml index 228f9db4f8ef2..1ffe0a97a2465 100644 --- a/.azure-pipelines/pr.yml +++ b/.azure-pipelines/pr.yml @@ -1,18 +1,14 @@ variables: - manylinux: false coverage: false -resources: - containers: - - container: manylinux1 - image: pyca/cryptography-manylinux1:x86_64 +pr: ['master', '3.9', '3.8', '3.7'] jobs: - job: Prebuild displayName: Pre-build checks pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./prebuild-checks.yml @@ -24,7 +20,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['docs.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 steps: - template: ./docs-steps.yml @@ -56,7 +52,7 @@ jobs: condition: and(succeeded(), eq(dependencies.Prebuild.outputs['tests.run'], 'true')) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(system.pullRequest.TargetBranch)-linux' @@ -69,37 +65,6 @@ jobs: dependencies: apt -- job: ManyLinux1_PR_Tests - displayName: ManyLinux1 PR Tests - dependsOn: Prebuild - condition: | - and( - and( - succeeded(), - eq(variables['manylinux'], 'true') - ), - eq(dependencies.Prebuild.outputs['tests.run'], 'true') - ) - - pool: - vmImage: ubuntu-16.04 - - container: manylinux1 - - variables: - testRunTitle: '$(system.pullRequest.TargetBranch)-manylinux1' - testRunPlatform: manylinux1 - openssl_version: '' - - steps: - - template: ./posix-steps.yml - parameters: - dependencies: yum - sudo_dependencies: '' - xvfb: false - patchcheck: false - - - job: Ubuntu_Coverage_PR_Tests displayName: Ubuntu PR Tests (coverage) dependsOn: Prebuild @@ -113,7 +78,7 @@ jobs: ) pool: - vmImage: ubuntu-16.04 + vmImage: ubuntu-18.04 variables: testRunTitle: '$(Build.SourceBranchName)-linux-coverage' From webhook-mailer at python.org Fri Aug 7 22:57:11 2020 From: webhook-mailer at python.org (Benjamin Kane) Date: Sat, 08 Aug 2020 02:57:11 -0000 Subject: [Python-checkins] Doc: Add a link to tutorial page from `open()` doc (GH-21737) Message-ID: https://github.com/python/cpython/commit/705f14556545699ab615ec98f707b438f9603767 commit: 705f14556545699ab615ec98f707b438f9603767 branch: master author: Benjamin Kane committer: GitHub date: 2020-08-08T11:57:03+09:00 summary: Doc: Add a link to tutorial page from `open()` doc (GH-21737) Adds a link to the "Reading and Writing Files" page so users can more easily discover how file handles are handled with the `with` context manager vs without it. files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 3c36b59befab9..43c47c1da9434 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1055,7 +1055,8 @@ are always available. They are listed here in alphabetical order. .. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) Open *file* and return a corresponding :term:`file object`. If the file - cannot be opened, an :exc:`OSError` is raised. + cannot be opened, an :exc:`OSError` is raised. See + :ref:`tut-files` for more examples of how to use this function. *file* is a :term:`path-like object` giving the pathname (absolute or relative to the current working directory) of the file to be opened or an From webhook-mailer at python.org Fri Aug 7 23:03:13 2020 From: webhook-mailer at python.org (Konge) Date: Sat, 08 Aug 2020 03:03:13 -0000 Subject: [Python-checkins] bpo-41497: Fix potential UnicodeDecodeError in dis CLI (GH-21757) Message-ID: https://github.com/python/cpython/commit/a4084b9d1e40c1c9259372263d1fe8c8a562b093 commit: a4084b9d1e40c1c9259372263d1fe8c8a562b093 branch: master author: Konge committer: GitHub date: 2020-08-08T12:03:09+09:00 summary: bpo-41497: Fix potential UnicodeDecodeError in dis CLI (GH-21757) files: A Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst M Lib/dis.py diff --git a/Lib/dis.py b/Lib/dis.py index 10e5f7fb08ab2..e289e176c78ff 100644 --- a/Lib/dis.py +++ b/Lib/dis.py @@ -542,7 +542,7 @@ def _test(): import argparse parser = argparse.ArgumentParser() - parser.add_argument('infile', type=argparse.FileType(), nargs='?', default='-') + parser.add_argument('infile', type=argparse.FileType('rb'), nargs='?', default='-') args = parser.parse_args() with args.infile as infile: source = infile.read() diff --git a/Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst b/Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst new file mode 100644 index 0000000000000..2c863ed7ffa3f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst @@ -0,0 +1 @@ +Fix potential UnicodeDecodeError in dis module. \ No newline at end of file From webhook-mailer at python.org Fri Aug 7 23:05:08 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 08 Aug 2020 03:05:08 -0000 Subject: [Python-checkins] Doc: Add a link to tutorial page from `open()` doc (GH-21737) Message-ID: https://github.com/python/cpython/commit/de5de413e7f03e9cdcaf0cf96b13f4fe6bd5ccb6 commit: de5de413e7f03e9cdcaf0cf96b13f4fe6bd5ccb6 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-07T20:05:04-07:00 summary: Doc: Add a link to tutorial page from `open()` doc (GH-21737) Adds a link to the "Reading and Writing Files" page so users can more easily discover how file handles are handled with the `with` context manager vs without it. (cherry picked from commit 705f14556545699ab615ec98f707b438f9603767) Co-authored-by: Benjamin Kane files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 0f7df0e5e9107..8cf755a0e6b38 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1035,7 +1035,8 @@ are always available. They are listed here in alphabetical order. .. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) Open *file* and return a corresponding :term:`file object`. If the file - cannot be opened, an :exc:`OSError` is raised. + cannot be opened, an :exc:`OSError` is raised. See + :ref:`tut-files` for more examples of how to use this function. *file* is a :term:`path-like object` giving the pathname (absolute or relative to the current working directory) of the file to be opened or an From webhook-mailer at python.org Fri Aug 7 23:21:41 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 08 Aug 2020 03:21:41 -0000 Subject: [Python-checkins] bpo-41497: Fix potential UnicodeDecodeError in dis CLI (GH-21757) Message-ID: https://github.com/python/cpython/commit/66c899661902edc18df96a5c3f22639310700491 commit: 66c899661902edc18df96a5c3f22639310700491 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-07T20:21:33-07:00 summary: bpo-41497: Fix potential UnicodeDecodeError in dis CLI (GH-21757) (cherry picked from commit a4084b9d1e40c1c9259372263d1fe8c8a562b093) Co-authored-by: Konge files: A Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst M Lib/dis.py diff --git a/Lib/dis.py b/Lib/dis.py index 10e5f7fb08ab2..e289e176c78ff 100644 --- a/Lib/dis.py +++ b/Lib/dis.py @@ -542,7 +542,7 @@ def _test(): import argparse parser = argparse.ArgumentParser() - parser.add_argument('infile', type=argparse.FileType(), nargs='?', default='-') + parser.add_argument('infile', type=argparse.FileType('rb'), nargs='?', default='-') args = parser.parse_args() with args.infile as infile: source = infile.read() diff --git a/Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst b/Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst new file mode 100644 index 0000000000000..2c863ed7ffa3f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-07-06-06-29.bpo-41497.aBtsWz.rst @@ -0,0 +1 @@ +Fix potential UnicodeDecodeError in dis module. \ No newline at end of file From webhook-mailer at python.org Sat Aug 8 05:32:50 2020 From: webhook-mailer at python.org (Hai Shi) Date: Sat, 08 Aug 2020 09:32:50 -0000 Subject: [Python-checkins] bpo-40275: Remove test helpers aliases in test.support (GH-21771) Message-ID: https://github.com/python/cpython/commit/d94af3f7ed98e6bfb4bf5f918f464b5e23d3ed1b commit: d94af3f7ed98e6bfb4bf5f918f464b5e23d3ed1b branch: master author: Hai Shi committer: GitHub date: 2020-08-08T11:32:41+02:00 summary: bpo-40275: Remove test helpers aliases in test.support (GH-21771) files: M Lib/ctypes/test/test_loading.py M Lib/test/support/__init__.py M Lib/test/test_os.py M Lib/test/test_posixpath.py diff --git a/Lib/ctypes/test/test_loading.py b/Lib/ctypes/test/test_loading.py index 38b45f95fefae..7b930f92c70cf 100644 --- a/Lib/ctypes/test/test_loading.py +++ b/Lib/ctypes/test/test_loading.py @@ -6,6 +6,7 @@ import unittest import test.support from test.support import import_helper +from test.support import os_helper from ctypes.util import find_library libc_name = None @@ -125,7 +126,7 @@ def test_load_dll_with_flags(self): else: ext = ".dll" - with test.support.temp_dir() as tmp: + with os_helper.temp_dir() as tmp: # We copy two files and load _sqlite3.dll (formerly .pyd), # which has a dependency on sqlite3.dll. Then we test # loading it in subprocesses to avoid it starting in memory diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index b517df7b53b6e..e9573d1335210 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -14,25 +14,6 @@ import types import unittest -from .import_helper import ( - CleanImport, DirsOnSysPath, _ignore_deprecated_imports, - _save_and_block_module, _save_and_remove_module, - forget, import_fresh_module, import_module, make_legacy_pyc, - modules_cleanup, modules_setup, unload) -from .os_helper import ( - FS_NONASCII, SAVEDCWD, TESTFN, TESTFN_ASCII, TESTFN_NONASCII, - TESTFN_UNENCODABLE, TESTFN_UNDECODABLE, - TESTFN_UNICODE, can_symlink, can_xattr, - change_cwd, create_empty_file, fd_count, - fs_is_case_insensitive, make_bad_fd, rmdir, - rmtree, skip_unless_symlink, skip_unless_xattr, - temp_cwd, temp_dir, temp_umask, unlink, - EnvironmentVarGuard, FakePath, _longpath) -from .warnings_helper import ( - WarningsRecorder, _filterwarnings, - check_no_resource_warning, check_no_warnings, - check_syntax_warning, check_warnings, ignore_warnings) - from .testresult import get_test_runner @@ -506,6 +487,7 @@ def check_syntax_error(testcase, statement, errtext='', *, lineno=None, offset=N def open_urlresource(url, *args, **kw): import urllib.request, urllib.parse + from .os_helper import unlink try: import gzip except ImportError: @@ -1326,6 +1308,8 @@ def skip_if_buggy_ucrt_strfptime(test): class PythonSymlink: """Creates a symlink for the current Python executable""" def __init__(self, link=None): + from .os_helper import TESTFN + self.link = link or os.path.abspath(TESTFN) self._linked = [] self.real = os.path.realpath(sys.executable) @@ -1980,6 +1964,7 @@ def skip_if_broken_multiprocessing_synchronize(): is no available semaphore implementation, or if creating a lock raises an OSError (on Linux only). """ + from .import_helper import import_module # Skip tests if the _multiprocessing extension is missing. import_module('_multiprocessing') diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index 03152072c1bf5..5126c84cf30c6 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -30,6 +30,7 @@ import uuid import warnings from test import support +from test.support import import_helper from test.support import os_helper from test.support import socket_helper from test.support import threading_helper @@ -2674,8 +2675,8 @@ def test_unlink_removes_junction(self): @unittest.skipUnless(sys.platform == "win32", "Win32 specific tests") class Win32NtTests(unittest.TestCase): def test_getfinalpathname_handles(self): - nt = support.import_module('nt') - ctypes = support.import_module('ctypes') + nt = import_helper.import_module('nt') + ctypes = import_helper.import_module('ctypes') import ctypes.wintypes kernel = ctypes.WinDLL('Kernel32.dll', use_last_error=True) diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py index f37e82505796d..42fd8ef8b1746 100644 --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -5,7 +5,7 @@ from test import test_genericpath from test.support import import_helper from test.support import os_helper -from test.support import FakePath +from test.support.os_helper import FakePath from unittest import mock try: From webhook-mailer at python.org Sat Aug 8 07:05:50 2020 From: webhook-mailer at python.org (Hai Shi) Date: Sat, 08 Aug 2020 11:05:50 -0000 Subject: [Python-checkins] bpo-40275: Use new test.support helper submodules in tests (GH-21785) Message-ID: https://github.com/python/cpython/commit/c6f282f3b1cb6da6febc3b8b6d2dc1ef714dbbf7 commit: c6f282f3b1cb6da6febc3b8b6d2dc1ef714dbbf7 branch: master author: Hai Shi committer: GitHub date: 2020-08-08T13:05:24+02:00 summary: bpo-40275: Use new test.support helper submodules in tests (GH-21785) files: M Lib/test/libregrtest/save_env.py M Lib/test/test__osx_support.py M Lib/test/test_dbm_ndbm.py M Lib/test/test_zipfile64.py diff --git a/Lib/test/libregrtest/save_env.py b/Lib/test/libregrtest/save_env.py index 50ed35364961c..4c9c692400b92 100644 --- a/Lib/test/libregrtest/save_env.py +++ b/Lib/test/libregrtest/save_env.py @@ -77,7 +77,7 @@ def get_urllib_requests__url_tempfiles(self): return list(urllib.request._url_tempfiles) def restore_urllib_requests__url_tempfiles(self, tempfiles): for filename in tempfiles: - support.unlink(filename) + os_helper.unlink(filename) def get_urllib_requests__opener(self): return urllib.request._opener @@ -245,9 +245,9 @@ def restore_files(self, saved_value): fn = os_helper.TESTFN if fn not in saved_value and (fn + '/') not in saved_value: if os.path.isfile(fn): - support.unlink(fn) + os_helper.unlink(fn) elif os.path.isdir(fn): - support.rmtree(fn) + os_helper.rmtree(fn) _lc = [getattr(locale, lc) for lc in dir(locale) if lc.startswith('LC_')] diff --git a/Lib/test/test__osx_support.py b/Lib/test/test__osx_support.py index 1a5d649b40f53..a3f41d2c5bd07 100644 --- a/Lib/test/test__osx_support.py +++ b/Lib/test/test__osx_support.py @@ -9,6 +9,7 @@ import unittest import test.support +from test.support import os_helper import _osx_support @@ -39,9 +40,9 @@ def test__find_executable(self): if self.env['PATH']: self.env['PATH'] = self.env['PATH'] + ':' self.env['PATH'] = self.env['PATH'] + os.path.abspath(self.temp_path_dir) - test.support.unlink(self.prog_name) + os_helper.unlink(self.prog_name) self.assertIsNone(_osx_support._find_executable(self.prog_name)) - self.addCleanup(test.support.unlink, self.prog_name) + self.addCleanup(os_helper.unlink, self.prog_name) with open(self.prog_name, 'w') as f: f.write("#!/bin/sh\n/bin/echo OK\n") os.chmod(self.prog_name, stat.S_IRWXU) @@ -52,8 +53,8 @@ def test__read_output(self): if self.env['PATH']: self.env['PATH'] = self.env['PATH'] + ':' self.env['PATH'] = self.env['PATH'] + os.path.abspath(self.temp_path_dir) - test.support.unlink(self.prog_name) - self.addCleanup(test.support.unlink, self.prog_name) + os_helper.unlink(self.prog_name) + self.addCleanup(os_helper.unlink, self.prog_name) with open(self.prog_name, 'w') as f: f.write("#!/bin/sh\n/bin/echo ExpectedOutput\n") os.chmod(self.prog_name, stat.S_IRWXU) @@ -143,8 +144,8 @@ def test__find_appropriate_compiler(self): suffix = (':' + self.env['PATH']) if self.env['PATH'] else '' self.env['PATH'] = os.path.abspath(self.temp_path_dir) + suffix for c_name, c_output in compilers: - test.support.unlink(c_name) - self.addCleanup(test.support.unlink, c_name) + os_helper.unlink(c_name) + self.addCleanup(os_helper.unlink, c_name) with open(c_name, 'w') as f: f.write("#!/bin/sh\n/bin/echo " + c_output) os.chmod(c_name, stat.S_IRWXU) @@ -221,8 +222,8 @@ def test__remove_unsupported_archs(self): suffix = (':' + self.env['PATH']) if self.env['PATH'] else '' self.env['PATH'] = os.path.abspath(self.temp_path_dir) + suffix c_name = 'clang' - test.support.unlink(c_name) - self.addCleanup(test.support.unlink, c_name) + os_helper.unlink(c_name) + self.addCleanup(os_helper.unlink, c_name) # exit status 255 means no PPC support in this compiler chain with open(c_name, 'w') as f: f.write("#!/bin/sh\nexit 255") diff --git a/Lib/test/test_dbm_ndbm.py b/Lib/test/test_dbm_ndbm.py index e17a1d9eca931..639c8330cd7b9 100644 --- a/Lib/test/test_dbm_ndbm.py +++ b/Lib/test/test_dbm_ndbm.py @@ -1,4 +1,3 @@ -from test import support from test.support import import_helper from test.support import os_helper import_helper.import_module("dbm.ndbm") #skip if not supported @@ -16,7 +15,7 @@ def setUp(self): def tearDown(self): for suffix in ['', '.pag', '.dir', '.db']: - support.unlink(self.filename + suffix) + os_helper.unlink(self.filename + suffix) def test_keys(self): self.d = dbm.ndbm.open(self.filename, 'c') @@ -108,7 +107,7 @@ def test_write_readonly_file(self): def test_nonascii_filename(self): filename = os_helper.TESTFN_NONASCII for suffix in ['', '.pag', '.dir', '.db']: - self.addCleanup(support.unlink, filename + suffix) + self.addCleanup(os_helper.unlink, filename + suffix) with dbm.ndbm.open(filename, 'c') as db: db[b'key'] = b'value' self.assertTrue(any(os.path.exists(filename + suffix) diff --git a/Lib/test/test_zipfile64.py b/Lib/test/test_zipfile64.py index 3a788de221264..810fdedef39dd 100644 --- a/Lib/test/test_zipfile64.py +++ b/Lib/test/test_zipfile64.py @@ -17,6 +17,7 @@ from tempfile import TemporaryFile +from test.support import os_helper from test.support import TESTFN, requires_zlib TESTFN2 = TESTFN + "2" @@ -138,8 +139,8 @@ def testMoreThan64kFilesAppend(self): self.assertEqual(content, "%d" % (i**3 % 57)) def tearDown(self): - support.unlink(TESTFN) - support.unlink(TESTFN2) + os_helper.unlink(TESTFN) + os_helper.unlink(TESTFN2) if __name__ == "__main__": unittest.main() From webhook-mailer at python.org Sat Aug 8 14:48:25 2020 From: webhook-mailer at python.org (Yaroslav Pankovych) Date: Sat, 08 Aug 2020 18:48:25 -0000 Subject: [Python-checkins] bpo-41455: Provide a link to how the third generation is collected in the GC docs (GH-21703) Message-ID: https://github.com/python/cpython/commit/82ca8fada15b121866530f2cdac1b7055be4a244 commit: 82ca8fada15b121866530f2cdac1b7055be4a244 branch: master author: Yaroslav Pankovych <31005942+P-Alban at users.noreply.github.com> committer: GitHub date: 2020-08-08T19:48:21+01:00 summary: bpo-41455: Provide a link to how the third generation is collected in the GC docs (GH-21703) Co-authored-by: Pablo Galindo files: M Doc/library/gc.rst diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 0c33c86530459..2d85cd3431711 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -106,9 +106,9 @@ The :mod:`gc` module provides the following functions: allocations minus the number of deallocations exceeds *threshold0*, collection starts. Initially only generation ``0`` is examined. If generation ``0`` has been examined more than *threshold1* times since generation ``1`` has been - examined, then generation ``1`` is examined as well. Similarly, *threshold2* - controls the number of collections of generation ``1`` before collecting - generation ``2``. + examined, then generation ``1`` is examined as well. + With the third generation, things are a bit more complicated, + see `Collecting the oldest generation `_ for more information. .. function:: get_count() From webhook-mailer at python.org Sat Aug 8 14:55:49 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 08 Aug 2020 18:55:49 -0000 Subject: [Python-checkins] bpo-41455: Provide a link to how the third generation is collected in the GC docs (GH-21703) Message-ID: https://github.com/python/cpython/commit/105cfb5b182da63e8481fcb009e92546d240c6b5 commit: 105cfb5b182da63e8481fcb009e92546d240c6b5 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-08T11:55:45-07:00 summary: bpo-41455: Provide a link to how the third generation is collected in the GC docs (GH-21703) Co-authored-by: Pablo Galindo (cherry picked from commit 82ca8fada15b121866530f2cdac1b7055be4a244) Co-authored-by: Yaroslav Pankovych <31005942+P-Alban at users.noreply.github.com> files: M Doc/library/gc.rst diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 13eda917b9a80..adaa30295b5e8 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -106,9 +106,9 @@ The :mod:`gc` module provides the following functions: allocations minus the number of deallocations exceeds *threshold0*, collection starts. Initially only generation ``0`` is examined. If generation ``0`` has been examined more than *threshold1* times since generation ``1`` has been - examined, then generation ``1`` is examined as well. Similarly, *threshold2* - controls the number of collections of generation ``1`` before collecting - generation ``2``. + examined, then generation ``1`` is examined as well. + With the third generation, things are a bit more complicated, + see `Collecting the oldest generation `_ for more information. .. function:: get_count() From webhook-mailer at python.org Sat Aug 8 15:29:11 2020 From: webhook-mailer at python.org (Jiajie Zhong) Date: Sat, 08 Aug 2020 19:29:11 -0000 Subject: [Python-checkins] Doc: Add output to example code in programming FAQ (GH-21346) Message-ID: https://github.com/python/cpython/commit/67acf74c4eaf64a860cc1bcda6efe6e9cb01f89b commit: 67acf74c4eaf64a860cc1bcda6efe6e9cb01f89b branch: master author: Jiajie Zhong committer: GitHub date: 2020-08-08T12:29:03-07:00 summary: Doc: Add output to example code in programming FAQ (GH-21346) Add output hint to document, part faq/programming, section [How do I write a function with output parameters (call by reference)?](https://docs.python.org/3/faq/programming.html#how-do-i-write-a-function-with-output-parameters-call-by-reference). This patch make the output hint just like prefix code block. files: M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 61ffc5dbdaa77..0731e92f6dbc6 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -518,14 +518,14 @@ desired effect in a number of ways. 1) By returning a tuple of the results:: - def func2(a, b): - a = 'new-value' # a and b are local names - b = b + 1 # assigned to new objects - return a, b # return new values - - x, y = 'old-value', 99 - x, y = func2(x, y) - print(x, y) # output: new-value 100 + >>> def func1(a, b): + ... a = 'new-value' # a and b are local names + ... b = b + 1 # assigned to new objects + ... return a, b # return new values + ... + >>> x, y = 'old-value', 99 + >>> func1(x, y) + ('new-value', 100) This is almost always the clearest solution. @@ -533,38 +533,41 @@ desired effect in a number of ways. 3) By passing a mutable (changeable in-place) object:: - def func1(a): - a[0] = 'new-value' # 'a' references a mutable list - a[1] = a[1] + 1 # changes a shared object - - args = ['old-value', 99] - func1(args) - print(args[0], args[1]) # output: new-value 100 + >>> def func2(a): + ... a[0] = 'new-value' # 'a' references a mutable list + ... a[1] = a[1] + 1 # changes a shared object + ... + >>> args = ['old-value', 99] + >>> func2(args) + >>> args + ['new-value', 100] 4) By passing in a dictionary that gets mutated:: - def func3(args): - args['a'] = 'new-value' # args is a mutable dictionary - args['b'] = args['b'] + 1 # change it in-place - - args = {'a': 'old-value', 'b': 99} - func3(args) - print(args['a'], args['b']) + >>> def func3(args): + ... args['a'] = 'new-value' # args is a mutable dictionary + ... args['b'] = args['b'] + 1 # change it in-place + ... + >>> args = {'a': 'old-value', 'b': 99} + >>> func3(args) + >>> args + {'a': 'new-value', 'b': 100} 5) Or bundle up values in a class instance:: - class callByRef: - def __init__(self, /, **args): - for key, value in args.items(): - setattr(self, key, value) - - def func4(args): - args.a = 'new-value' # args is a mutable callByRef - args.b = args.b + 1 # change object in-place - - args = callByRef(a='old-value', b=99) - func4(args) - print(args.a, args.b) + >>> class Namespace: + ... def __init__(self, /, **args): + ... for key, value in args.items(): + ... setattr(self, key, value) + ... + >>> def func4(args): + ... args.a = 'new-value' # args is a mutable Namespace + ... args.b = args.b + 1 # change object in-place + ... + >>> args = Namespace(a='old-value', b=99) + >>> func4(args) + >>> vars(args) + {'a': 'new-value', 'b': 100} There's almost never a good reason to get this complicated. From webhook-mailer at python.org Sun Aug 9 06:51:21 2020 From: webhook-mailer at python.org (Zackery Spytz) Date: Sun, 09 Aug 2020 10:51:21 -0000 Subject: [Python-checkins] bpo-35018: Sax parser should provide user access to lexical handlers (GH-20958) Message-ID: https://github.com/python/cpython/commit/e28b8c93878072dc02b116108ef5443084290d47 commit: e28b8c93878072dc02b116108ef5443084290d47 branch: master author: Zackery Spytz committer: GitHub date: 2020-08-09T12:50:53+02:00 summary: bpo-35018: Sax parser should provide user access to lexical handlers (GH-20958) Co-Authored-By: Jonathan Gossage files: A Misc/NEWS.d/next/Library/2020-06-17-23-49-45.bpo-35018.NP5_Qk.rst M Doc/library/xml.sax.handler.rst M Doc/whatsnew/3.10.rst M Lib/test/test_sax.py M Lib/xml/sax/handler.py diff --git a/Doc/library/xml.sax.handler.rst b/Doc/library/xml.sax.handler.rst index ae0877ca90db0..3746a58c9b955 100644 --- a/Doc/library/xml.sax.handler.rst +++ b/Doc/library/xml.sax.handler.rst @@ -11,12 +11,12 @@ -------------- -The SAX API defines four kinds of handlers: content handlers, DTD handlers, -error handlers, and entity resolvers. Applications normally only need to -implement those interfaces whose events they are interested in; they can -implement the interfaces in a single object or in multiple objects. Handler -implementations should inherit from the base classes provided in the module -:mod:`xml.sax.handler`, so that all methods get default implementations. +The SAX API defines five kinds of handlers: content handlers, DTD handlers, +error handlers, entity resolvers and lexical handlers. Applications normally +only need to implement those interfaces whose events they are interested in; +they can implement the interfaces in a single object or in multiple objects. +Handler implementations should inherit from the base classes provided in the +module :mod:`xml.sax.handler`, so that all methods get default implementations. .. class:: ContentHandler @@ -47,6 +47,12 @@ implementations should inherit from the base classes provided in the module application. The methods of this object control whether errors are immediately converted to exceptions or are handled in some other way. + +.. class:: LexicalHandler + + Interface used by the parser to represent low freqency events which may not + be of interest to many applications. + In addition to these classes, :mod:`xml.sax.handler` provides symbolic constants for the feature and property names. @@ -114,7 +120,7 @@ for the feature and property names. .. data:: property_lexical_handler | value: ``"http://xml.org/sax/properties/lexical-handler"`` - | data type: xml.sax.sax2lib.LexicalHandler (not supported in Python 2) + | data type: xml.sax.handler.LexicalHandler (not supported in Python 2) | description: An optional extension handler for lexical events like comments. | access: read/write @@ -413,3 +419,45 @@ the passed-in exception object. information will continue to be passed to the application. Raising an exception in this method will cause parsing to end. + +.. _lexical-handler-objects: + +LexicalHandler Objects +---------------------- +Optional SAX2 handler for lexical events. + +This handler is used to obtain lexical information about an XML +document. Lexical information includes information describing the +document encoding used and XML comments embedded in the document, as +well as section boundaries for the DTD and for any CDATA sections. +The lexical handlers are used in the same manner as content handlers. + +Set the LexicalHandler of an XMLReader by using the setProperty method +with the property identifier +``'http://xml.org/sax/properties/lexical-handler'``. + + +.. method:: LexicalHandler.comment(content) + + Reports a comment anywhere in the document (including the DTD and + outside the document element). + +.. method:: LexicalHandler.startDTD(name, public_id, system_id) + + Reports the start of the DTD declarations if the document has an + associated DTD. + +.. method:: LexicalHandler.endDTD() + + Reports the end of DTD declaration. + +.. method:: LexicalHandler.startCDATA() + + Reports the start of a CDATA marked section. + + The contents of the CDATA marked section will be reported through + the characters handler. + +.. method:: LexicalHandler.endCDATA() + + Reports the end of a CDATA marked section. diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 62bb1438416e6..2af0ea3f4dd64 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -139,6 +139,13 @@ Add :data:`sys.orig_argv` attribute: the list of the original command line arguments passed to the Python executable. (Contributed by Victor Stinner in :issue:`23427`.) +xml +--- + +Add a :class:`~xml.sax.handler.LexicalHandler` class to the +:mod:`xml.sax.handler` module. +(Contributed by Jonathan Gossage and Zackery Spytz in :issue:`35018`.) + Optimizations ============= diff --git a/Lib/test/test_sax.py b/Lib/test/test_sax.py index cfc674be5d141..801143f9b5f81 100644 --- a/Lib/test/test_sax.py +++ b/Lib/test/test_sax.py @@ -13,7 +13,8 @@ from xml.sax.saxutils import XMLGenerator, escape, unescape, quoteattr, \ XMLFilterBase, prepare_input_source from xml.sax.expatreader import create_parser -from xml.sax.handler import feature_namespaces, feature_external_ges +from xml.sax.handler import (feature_namespaces, feature_external_ges, + LexicalHandler) from xml.sax.xmlreader import InputSource, AttributesImpl, AttributesNSImpl from io import BytesIO, StringIO import codecs @@ -1356,6 +1357,155 @@ def test_nsattrs_wattr(self): self.assertEqual(attrs.getQNameByName((ns_uri, "attr")), "ns:attr") +class LexicalHandlerTest(unittest.TestCase): + def setUp(self): + self.parser = None + + self.specified_version = '1.0' + self.specified_encoding = 'UTF-8' + self.specified_doctype = 'wish' + self.specified_entity_names = ('nbsp', 'source', 'target') + self.specified_comment = ('Comment in a DTD', + 'Really! You think so?') + self.test_data = StringIO() + self.test_data.write('\n'. + format(self.specified_version, + self.specified_encoding)) + self.test_data.write('\n'. + format(self.specified_comment[0])) + self.test_data.write('\n'. + format(self.specified_doctype)) + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.write('\n'. + format(self.specified_entity_names[0])) + self.test_data.write('\n'. + format(self.specified_entity_names[1])) + self.test_data.write('\n'. + format(self.specified_entity_names[2])) + self.test_data.write(']>\n') + self.test_data.write('<{}>'.format(self.specified_doctype)) + self.test_data.write('Aristotle\n') + self.test_data.write('Alexander\n') + self.test_data.write('Supplication\n') + self.test_data.write('Teach me patience!\n') + self.test_data.write('
&{};&{};&{};
\n'. + format(self.specified_entity_names[1], + self.specified_entity_names[0], + self.specified_entity_names[2])) + self.test_data.write('\n'.format(self.specified_comment[1])) + self.test_data.write('\n'.format(self.specified_doctype)) + self.test_data.seek(0) + + # Data received from handlers - to be validated + self.version = None + self.encoding = None + self.standalone = None + self.doctype = None + self.publicID = None + self.systemID = None + self.end_of_dtd = False + self.comments = [] + + def test_handlers(self): + class TestLexicalHandler(LexicalHandler): + def __init__(self, test_harness, *args, **kwargs): + super().__init__(*args, **kwargs) + self.test_harness = test_harness + + def startDTD(self, doctype, publicID, systemID): + self.test_harness.doctype = doctype + self.test_harness.publicID = publicID + self.test_harness.systemID = systemID + + def endDTD(self): + self.test_harness.end_of_dtd = True + + def comment(self, text): + self.test_harness.comments.append(text) + + self.parser = create_parser() + self.parser.setContentHandler(ContentHandler()) + self.parser.setProperty( + 'http://xml.org/sax/properties/lexical-handler', + TestLexicalHandler(self)) + source = InputSource() + source.setCharacterStream(self.test_data) + self.parser.parse(source) + self.assertEqual(self.doctype, self.specified_doctype) + self.assertIsNone(self.publicID) + self.assertIsNone(self.systemID) + self.assertTrue(self.end_of_dtd) + self.assertEqual(len(self.comments), + len(self.specified_comment)) + self.assertEqual(f' {self.specified_comment[0]} ', self.comments[0]) + + +class CDATAHandlerTest(unittest.TestCase): + def setUp(self): + self.parser = None + self.specified_chars = [] + self.specified_chars.append(('Parseable character data', False)) + self.specified_chars.append(('<> &% - assorted other XML junk.', True)) + self.char_index = 0 # Used to index specified results within handlers + self.test_data = StringIO() + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.write(f'{self.specified_chars[0][0]}\n') + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.write(f'\n') + self.test_data.write('\n') + self.test_data.write('\n') + self.test_data.seek(0) + + # Data received from handlers - to be validated + self.chardata = [] + self.in_cdata = False + + def test_handlers(self): + class TestLexicalHandler(LexicalHandler): + def __init__(self, test_harness, *args, **kwargs): + super().__init__(*args, **kwargs) + self.test_harness = test_harness + + def startCDATA(self): + self.test_harness.in_cdata = True + + def endCDATA(self): + self.test_harness.in_cdata = False + + class TestCharHandler(ContentHandler): + def __init__(self, test_harness, *args, **kwargs): + super().__init__(*args, **kwargs) + self.test_harness = test_harness + + def characters(self, content): + if content != '\n': + h = self.test_harness + t = h.specified_chars[h.char_index] + h.assertEqual(t[0], content) + h.assertEqual(t[1], h.in_cdata) + h.char_index += 1 + + self.parser = create_parser() + self.parser.setContentHandler(TestCharHandler(self)) + self.parser.setProperty( + 'http://xml.org/sax/properties/lexical-handler', + TestLexicalHandler(self)) + source = InputSource() + source.setCharacterStream(self.test_data) + self.parser.parse(source) + + self.assertFalse(self.in_cdata) + self.assertEqual(self.char_index, 2) + + def test_main(): run_unittest(MakeParserTest, ParseTest, @@ -1368,7 +1518,10 @@ def test_main(): StreamReaderWriterXmlgenTest, ExpatReaderTest, ErrorReportingTest, - XmlReaderTest) + XmlReaderTest, + LexicalHandlerTest, + CDATAHandlerTest) + if __name__ == "__main__": test_main() diff --git a/Lib/xml/sax/handler.py b/Lib/xml/sax/handler.py index 481733d2cbe6e..e8d417e519423 100644 --- a/Lib/xml/sax/handler.py +++ b/Lib/xml/sax/handler.py @@ -340,3 +340,48 @@ def resolveEntity(self, publicId, systemId): property_xml_string, property_encoding, property_interning_dict] + + +class LexicalHandler: + """Optional SAX2 handler for lexical events. + + This handler is used to obtain lexical information about an XML + document, that is, information about how the document was encoded + (as opposed to what it contains, which is reported to the + ContentHandler), such as comments and CDATA marked section + boundaries. + + To set the LexicalHandler of an XMLReader, use the setProperty + method with the property identifier + 'http://xml.org/sax/properties/lexical-handler'.""" + + def comment(self, content): + """Reports a comment anywhere in the document (including the + DTD and outside the document element). + + content is a string that holds the contents of the comment.""" + + def startDTD(self, name, public_id, system_id): + """Report the start of the DTD declarations, if the document + has an associated DTD. + + A startEntity event will be reported before declaration events + from the external DTD subset are reported, and this can be + used to infer from which subset DTD declarations derive. + + name is the name of the document element type, public_id the + public identifier of the DTD (or None if none were supplied) + and system_id the system identfier of the external subset (or + None if none were supplied).""" + + def endDTD(self): + """Signals the end of DTD declarations.""" + + def startCDATA(self): + """Reports the beginning of a CDATA marked section. + + The contents of the CDATA marked section will be reported + through the characters event.""" + + def endCDATA(self): + """Reports the end of a CDATA marked section.""" diff --git a/Misc/NEWS.d/next/Library/2020-06-17-23-49-45.bpo-35018.NP5_Qk.rst b/Misc/NEWS.d/next/Library/2020-06-17-23-49-45.bpo-35018.NP5_Qk.rst new file mode 100644 index 0000000000000..f764323ae631c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-06-17-23-49-45.bpo-35018.NP5_Qk.rst @@ -0,0 +1,2 @@ +Add the :class:`xml.sax.handler.LexicalHandler` class that is present in +other SAX XML implementations. From webhook-mailer at python.org Sun Aug 9 13:08:26 2020 From: webhook-mailer at python.org (Terry Jan Reedy) Date: Sun, 09 Aug 2020 17:08:26 -0000 Subject: [Python-checkins] Improve renamed test_run.RecursionLimitTest (GH-21794) Message-ID: https://github.com/python/cpython/commit/8b67bf907c51846853127176cbb2982d102a2c2d commit: 8b67bf907c51846853127176cbb2982d102a2c2d branch: master author: Terry Jan Reedy committer: GitHub date: 2020-08-09T13:08:19-04:00 summary: Improve renamed test_run.RecursionLimitTest (GH-21794) PEP 8 style and new comments. files: M Lib/idlelib/idle_test/test_run.py diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index 9995dbe2eca50..e2bdf1cfee352 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -282,7 +282,8 @@ def test_close(self): self.assertRaises(TypeError, f.close, 1) -class TestSysRecursionLimitWrappers(unittest.TestCase): +class RecursionLimitTest(unittest.TestCase): + # Test (un)install_recursionlimit_wrappers and fixdoc. def test_bad_setrecursionlimit_calls(self): run.install_recursionlimit_wrappers() @@ -296,12 +297,12 @@ def test_roundtrip(self): run.install_recursionlimit_wrappers() self.addCleanup(run.uninstall_recursionlimit_wrappers) - # check that setting the recursion limit works + # Check that setting the recursion limit works. orig_reclimit = sys.getrecursionlimit() self.addCleanup(sys.setrecursionlimit, orig_reclimit) sys.setrecursionlimit(orig_reclimit + 3) - # check that the new limit is returned by sys.getrecursionlimit() + # Check that the new limit is returned by sys.getrecursionlimit(). new_reclimit = sys.getrecursionlimit() self.assertEqual(new_reclimit, orig_reclimit + 3) @@ -313,6 +314,7 @@ def test_default_recursion_limit_preserved(self): self.assertEqual(new_reclimit, orig_reclimit) def test_fixdoc(self): + # Put here until better place for miscellaneous test. def func(): "docstring" run.fixdoc(func, "more") self.assertEqual(func.__doc__, "docstring\n\nmore") From webhook-mailer at python.org Sun Aug 9 13:36:03 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 09 Aug 2020 17:36:03 -0000 Subject: [Python-checkins] Improve renamed test_run.RecursionLimitTest (GH-21794) Message-ID: https://github.com/python/cpython/commit/692552388d0d89523416eac24f436dafa2bb4914 commit: 692552388d0d89523416eac24f436dafa2bb4914 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-09T10:35:59-07:00 summary: Improve renamed test_run.RecursionLimitTest (GH-21794) PEP 8 style and new comments. (cherry picked from commit 8b67bf907c51846853127176cbb2982d102a2c2d) Co-authored-by: Terry Jan Reedy files: M Lib/idlelib/idle_test/test_run.py diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index 9995dbe2eca50..e2bdf1cfee352 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -282,7 +282,8 @@ def test_close(self): self.assertRaises(TypeError, f.close, 1) -class TestSysRecursionLimitWrappers(unittest.TestCase): +class RecursionLimitTest(unittest.TestCase): + # Test (un)install_recursionlimit_wrappers and fixdoc. def test_bad_setrecursionlimit_calls(self): run.install_recursionlimit_wrappers() @@ -296,12 +297,12 @@ def test_roundtrip(self): run.install_recursionlimit_wrappers() self.addCleanup(run.uninstall_recursionlimit_wrappers) - # check that setting the recursion limit works + # Check that setting the recursion limit works. orig_reclimit = sys.getrecursionlimit() self.addCleanup(sys.setrecursionlimit, orig_reclimit) sys.setrecursionlimit(orig_reclimit + 3) - # check that the new limit is returned by sys.getrecursionlimit() + # Check that the new limit is returned by sys.getrecursionlimit(). new_reclimit = sys.getrecursionlimit() self.assertEqual(new_reclimit, orig_reclimit + 3) @@ -313,6 +314,7 @@ def test_default_recursion_limit_preserved(self): self.assertEqual(new_reclimit, orig_reclimit) def test_fixdoc(self): + # Put here until better place for miscellaneous test. def func(): "docstring" run.fixdoc(func, "more") self.assertEqual(func.__doc__, "docstring\n\nmore") From webhook-mailer at python.org Sun Aug 9 14:54:52 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 09 Aug 2020 18:54:52 -0000 Subject: [Python-checkins] [3.8] Doc: Add output to example code in programming FAQ (GH-21346) (GH-21791) Message-ID: https://github.com/python/cpython/commit/f421865c76a34a80c57350bcfbde664f4e80313d commit: f421865c76a34a80c57350bcfbde664f4e80313d branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-09T11:54:33-07:00 summary: [3.8] Doc: Add output to example code in programming FAQ (GH-21346) (GH-21791) Add output hint to document, part faq/programming, section [How do I write a function with output parameters (call by reference)?](https://docs.python.org/3/faq/programming.htmlGH-how-do-i-write-a-function-with-output-parameters-call-by-reference). This patch make the output hint just like prefix code block. (cherry picked from commit 67acf74c4eaf64a860cc1bcda6efe6e9cb01f89b) Co-authored-by: Jiajie Zhong Automerge-Triggered-By: @merwok files: M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 6cc1b52ed7b98..81048c9422f32 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -518,14 +518,14 @@ desired effect in a number of ways. 1) By returning a tuple of the results:: - def func2(a, b): - a = 'new-value' # a and b are local names - b = b + 1 # assigned to new objects - return a, b # return new values - - x, y = 'old-value', 99 - x, y = func2(x, y) - print(x, y) # output: new-value 100 + >>> def func1(a, b): + ... a = 'new-value' # a and b are local names + ... b = b + 1 # assigned to new objects + ... return a, b # return new values + ... + >>> x, y = 'old-value', 99 + >>> func1(x, y) + ('new-value', 100) This is almost always the clearest solution. @@ -533,38 +533,41 @@ desired effect in a number of ways. 3) By passing a mutable (changeable in-place) object:: - def func1(a): - a[0] = 'new-value' # 'a' references a mutable list - a[1] = a[1] + 1 # changes a shared object - - args = ['old-value', 99] - func1(args) - print(args[0], args[1]) # output: new-value 100 + >>> def func2(a): + ... a[0] = 'new-value' # 'a' references a mutable list + ... a[1] = a[1] + 1 # changes a shared object + ... + >>> args = ['old-value', 99] + >>> func2(args) + >>> args + ['new-value', 100] 4) By passing in a dictionary that gets mutated:: - def func3(args): - args['a'] = 'new-value' # args is a mutable dictionary - args['b'] = args['b'] + 1 # change it in-place - - args = {'a': 'old-value', 'b': 99} - func3(args) - print(args['a'], args['b']) + >>> def func3(args): + ... args['a'] = 'new-value' # args is a mutable dictionary + ... args['b'] = args['b'] + 1 # change it in-place + ... + >>> args = {'a': 'old-value', 'b': 99} + >>> func3(args) + >>> args + {'a': 'new-value', 'b': 100} 5) Or bundle up values in a class instance:: - class callByRef: - def __init__(self, /, **args): - for key, value in args.items(): - setattr(self, key, value) - - def func4(args): - args.a = 'new-value' # args is a mutable callByRef - args.b = args.b + 1 # change object in-place - - args = callByRef(a='old-value', b=99) - func4(args) - print(args.a, args.b) + >>> class Namespace: + ... def __init__(self, /, **args): + ... for key, value in args.items(): + ... setattr(self, key, value) + ... + >>> def func4(args): + ... args.a = 'new-value' # args is a mutable Namespace + ... args.b = args.b + 1 # change object in-place + ... + >>> args = Namespace(a='old-value', b=99) + >>> func4(args) + >>> vars(args) + {'a': 'new-value', 'b': 100} There's almost never a good reason to get this complicated. From webhook-mailer at python.org Sun Aug 9 16:08:38 2020 From: webhook-mailer at python.org (Terry Jan Reedy) Date: Sun, 09 Aug 2020 20:08:38 -0000 Subject: [Python-checkins] bpo-41468: Improve and test IDLE run error exit (GH-21798) Message-ID: https://github.com/python/cpython/commit/f2e161c27964a59bc5ab20d96f87ba5862c6222d commit: f2e161c27964a59bc5ab20d96f87ba5862c6222d branch: master author: Terry Jan Reedy committer: GitHub date: 2020-08-09T16:08:30-04:00 summary: bpo-41468: Improve and test IDLE run error exit (GH-21798) A message box pops up when an unexpected error stops the run process. Tell users it is likely a random glitch, but report it if not. files: A Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst M Lib/idlelib/NEWS.txt M Lib/idlelib/idle_test/test_run.py M Lib/idlelib/run.py diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index e0a671983c746..fd762077b1b3c 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2020-10-05? ====================================== +bpo-41468: Improve IDLE run crash error message (which users should +never see). + bpo-41373: Save files loaded with no line ending, as when blank, or different line endings, by setting its line ending to the system default. Fix regression in 3.8.4 and 3.9.0b4. diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index e2bdf1cfee352..469c13d756d5e 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -1,9 +1,10 @@ -"Test run, coverage 42%." +"Test run, coverage 49%." from idlelib import run import unittest from unittest import mock -from test.support import captured_stderr +from idlelib.idle_test.mock_idle import Func +from test.support import captured_output, captured_stderr import io import sys @@ -323,5 +324,32 @@ def func(): "docstring" self.assertEqual(func.__doc__, "more") +class HandleErrorTest(unittest.TestCase): + # Method of MyRPCServer + func = Func() + @mock.patch('idlelib.run.thread.interrupt_main', new=func) + def test_error(self): + eq = self.assertEqual + with captured_output('__stderr__') as err: + try: + raise EOFError + except EOFError: + run.MyRPCServer.handle_error(None, 'abc', '123') + eq(run.exit_now, True) + run.exit_now = False + eq(err.getvalue(), '') + + try: + raise IndexError + except IndexError: + run.MyRPCServer.handle_error(None, 'abc', '123') + eq(run.quitting, True) + run.quitting = False + msg = err.getvalue() + self.assertIn('abc', msg) + self.assertIn('123', msg) + self.assertIn('IndexError', msg) + eq(self.func.called, 2) + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py index 5bd84aadcd801..1e84ecc6584ef 100644 --- a/Lib/idlelib/run.py +++ b/Lib/idlelib/run.py @@ -387,14 +387,21 @@ def handle_error(self, request, client_address): thread.interrupt_main() except: erf = sys.__stderr__ - print('\n' + '-'*40, file=erf) - print('Unhandled server exception!', file=erf) - print('Thread: %s' % threading.current_thread().name, file=erf) - print('Client Address: ', client_address, file=erf) - print('Request: ', repr(request), file=erf) - traceback.print_exc(file=erf) - print('\n*** Unrecoverable, server exiting!', file=erf) - print('-'*40, file=erf) + print(textwrap.dedent(f""" + {'-'*40} + Unhandled exception in user code execution server!' + Thread: {threading.current_thread().name} + IDLE Client Address: {client_address} + Request: {request!r} + """), file=erf) + traceback.print_exc(limit=-20, file=erf) + print(textwrap.dedent(f""" + *** Unrecoverable, server exiting! + + Users should never see this message; it is likely transient. + If this recurs, report this with a copy of the message + and an explanation of how to make it repeat. + {'-'*40}"""), file=erf) quitting = True thread.interrupt_main() diff --git a/Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst b/Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst new file mode 100644 index 0000000000000..e41c7d574905c --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst @@ -0,0 +1 @@ +Improve IDLE run crash error message (which users should never see). From webhook-mailer at python.org Sun Aug 9 16:26:41 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 09 Aug 2020 20:26:41 -0000 Subject: [Python-checkins] bpo-41468: Improve and test IDLE run error exit (GH-21798) Message-ID: https://github.com/python/cpython/commit/61f23cb62d6bdd72b61fc36abf4c1492493d71af commit: 61f23cb62d6bdd72b61fc36abf4c1492493d71af branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-09T13:26:37-07:00 summary: bpo-41468: Improve and test IDLE run error exit (GH-21798) A message box pops up when an unexpected error stops the run process. Tell users it is likely a random glitch, but report it if not. (cherry picked from commit f2e161c27964a59bc5ab20d96f87ba5862c6222d) Co-authored-by: Terry Jan Reedy files: A Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst M Lib/idlelib/NEWS.txt M Lib/idlelib/idle_test/test_run.py M Lib/idlelib/run.py diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt index 41ff15d84eac7..f395f0c72f662 100644 --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -3,6 +3,9 @@ Released on 2020-09-14? ====================================== +bpo-41468: Improve IDLE run crash error message (which users should +never see). + bpo-41373: Save files loaded with no line ending, as when blank, or different line endings, by setting its line ending to the system default. Fix regression in 3.8.4 and 3.9.0b4. diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index e2bdf1cfee352..469c13d756d5e 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -1,9 +1,10 @@ -"Test run, coverage 42%." +"Test run, coverage 49%." from idlelib import run import unittest from unittest import mock -from test.support import captured_stderr +from idlelib.idle_test.mock_idle import Func +from test.support import captured_output, captured_stderr import io import sys @@ -323,5 +324,32 @@ def func(): "docstring" self.assertEqual(func.__doc__, "more") +class HandleErrorTest(unittest.TestCase): + # Method of MyRPCServer + func = Func() + @mock.patch('idlelib.run.thread.interrupt_main', new=func) + def test_error(self): + eq = self.assertEqual + with captured_output('__stderr__') as err: + try: + raise EOFError + except EOFError: + run.MyRPCServer.handle_error(None, 'abc', '123') + eq(run.exit_now, True) + run.exit_now = False + eq(err.getvalue(), '') + + try: + raise IndexError + except IndexError: + run.MyRPCServer.handle_error(None, 'abc', '123') + eq(run.quitting, True) + run.quitting = False + msg = err.getvalue() + self.assertIn('abc', msg) + self.assertIn('123', msg) + self.assertIn('IndexError', msg) + eq(self.func.called, 2) + if __name__ == '__main__': unittest.main(verbosity=2) diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py index 5bd84aadcd801..1e84ecc6584ef 100644 --- a/Lib/idlelib/run.py +++ b/Lib/idlelib/run.py @@ -387,14 +387,21 @@ def handle_error(self, request, client_address): thread.interrupt_main() except: erf = sys.__stderr__ - print('\n' + '-'*40, file=erf) - print('Unhandled server exception!', file=erf) - print('Thread: %s' % threading.current_thread().name, file=erf) - print('Client Address: ', client_address, file=erf) - print('Request: ', repr(request), file=erf) - traceback.print_exc(file=erf) - print('\n*** Unrecoverable, server exiting!', file=erf) - print('-'*40, file=erf) + print(textwrap.dedent(f""" + {'-'*40} + Unhandled exception in user code execution server!' + Thread: {threading.current_thread().name} + IDLE Client Address: {client_address} + Request: {request!r} + """), file=erf) + traceback.print_exc(limit=-20, file=erf) + print(textwrap.dedent(f""" + *** Unrecoverable, server exiting! + + Users should never see this message; it is likely transient. + If this recurs, report this with a copy of the message + and an explanation of how to make it repeat. + {'-'*40}"""), file=erf) quitting = True thread.interrupt_main() diff --git a/Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst b/Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst new file mode 100644 index 0000000000000..e41c7d574905c --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2020-08-09-13-42-55.bpo-41468.zkP0_Y.rst @@ -0,0 +1 @@ +Improve IDLE run crash error message (which users should never see). From webhook-mailer at python.org Mon Aug 10 09:44:04 2020 From: webhook-mailer at python.org (Terry Jan Reedy) Date: Mon, 10 Aug 2020 13:44:04 -0000 Subject: [Python-checkins] bpo-41514: Fix buggy IDLE test (GH-21808) Message-ID: https://github.com/python/cpython/commit/416f0b71ba84fe83ee2ba4399b8a28712702980b commit: 416f0b71ba84fe83ee2ba4399b8a28712702980b branch: master author: Terry Jan Reedy committer: GitHub date: 2020-08-10T06:43:56-07:00 summary: bpo-41514: Fix buggy IDLE test (GH-21808) test_run method test_fatal_error failed when run twice, as with python -m test -m test_fatal_error test_idle test_idle because func.called was not reinitialized to 0. This bug caused a failure on a refleak buildbot. files: M Lib/idlelib/idle_test/test_run.py diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index 469c13d756d5e..37c0d4525e56c 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -326,11 +326,11 @@ def func(): "docstring" class HandleErrorTest(unittest.TestCase): # Method of MyRPCServer - func = Func() - @mock.patch('idlelib.run.thread.interrupt_main', new=func) - def test_error(self): + def test_fatal_error(self): eq = self.assertEqual - with captured_output('__stderr__') as err: + with captured_output('__stderr__') as err,\ + mock.patch('idlelib.run.thread.interrupt_main', + new_callable=Func) as func: try: raise EOFError except EOFError: @@ -349,7 +349,7 @@ def test_error(self): self.assertIn('abc', msg) self.assertIn('123', msg) self.assertIn('IndexError', msg) - eq(self.func.called, 2) + eq(func.called, 2) if __name__ == '__main__': unittest.main(verbosity=2) From webhook-mailer at python.org Mon Aug 10 10:01:24 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Aug 2020 14:01:24 -0000 Subject: [Python-checkins] bpo-41514: Fix buggy IDLE test (GH-21808) Message-ID: https://github.com/python/cpython/commit/860bc0ea70c365825bfd9b7de7685cf6842ca3c7 commit: 860bc0ea70c365825bfd9b7de7685cf6842ca3c7 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-10T07:01:14-07:00 summary: bpo-41514: Fix buggy IDLE test (GH-21808) test_run method test_fatal_error failed when run twice, as with python -m test -m test_fatal_error test_idle test_idle because func.called was not reinitialized to 0. This bug caused a failure on a refleak buildbot. (cherry picked from commit 416f0b71ba84fe83ee2ba4399b8a28712702980b) Co-authored-by: Terry Jan Reedy files: M Lib/idlelib/idle_test/test_run.py diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py index 469c13d756d5e..37c0d4525e56c 100644 --- a/Lib/idlelib/idle_test/test_run.py +++ b/Lib/idlelib/idle_test/test_run.py @@ -326,11 +326,11 @@ def func(): "docstring" class HandleErrorTest(unittest.TestCase): # Method of MyRPCServer - func = Func() - @mock.patch('idlelib.run.thread.interrupt_main', new=func) - def test_error(self): + def test_fatal_error(self): eq = self.assertEqual - with captured_output('__stderr__') as err: + with captured_output('__stderr__') as err,\ + mock.patch('idlelib.run.thread.interrupt_main', + new_callable=Func) as func: try: raise EOFError except EOFError: @@ -349,7 +349,7 @@ def test_error(self): self.assertIn('abc', msg) self.assertIn('123', msg) self.assertIn('IndexError', msg) - eq(self.func.called, 2) + eq(func.called, 2) if __name__ == '__main__': unittest.main(verbosity=2) From webhook-mailer at python.org Mon Aug 10 10:32:30 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Mon, 10 Aug 2020 14:32:30 -0000 Subject: [Python-checkins] bpo-41324 Add a minimal decimal capsule API (#21519) Message-ID: https://github.com/python/cpython/commit/39042e00ab01d6521548c1b7cc6554c09f4389ff commit: 39042e00ab01d6521548c1b7cc6554c09f4389ff branch: master author: Stefan Krah committer: GitHub date: 2020-08-10T16:32:21+02:00 summary: bpo-41324 Add a minimal decimal capsule API (#21519) files: A Doc/c-api/decimal.rst A Include/pydecimal.h A Misc/NEWS.d/next/C API/2020-08-10-16-05-08.bpo-41324.waZD35.rst M Doc/c-api/concrete.rst M Lib/test/test_decimal.py M Modules/_decimal/_decimal.c M Modules/_decimal/tests/deccheck.py M Modules/_testcapimodule.c diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst index c1d9fa1b41a3f..bf263d6e4c264 100644 --- a/Doc/c-api/concrete.rst +++ b/Doc/c-api/concrete.rst @@ -115,3 +115,4 @@ Other Objects coro.rst contextvars.rst datetime.rst + decimal.rst diff --git a/Doc/c-api/decimal.rst b/Doc/c-api/decimal.rst new file mode 100644 index 0000000000000..f530571ebae57 --- /dev/null +++ b/Doc/c-api/decimal.rst @@ -0,0 +1,231 @@ +.. sectionauthor:: Stefan Krah + +.. highlight:: c + + +Decimal capsule API +=================== + +Capsule API functions can be used in the same manner as regular library +functions, provided that the API has been initialized. + + +Initialize +---------- + +Typically, a C extension module that uses the decimal API will do these +steps in its init function: + +.. code-block:: + + #include "pydecimal.h" + + static int decimal_initialized = 0; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + +Type checking, predicates, accessors +------------------------------------ + +.. c:function:: int PyDec_TypeCheck(const PyObject *dec) + + Return 1 if ``dec`` is a Decimal, 0 otherwise. This function does not set + any exceptions. + + +.. c:function:: int PyDec_IsSpecial(const PyObject *dec) + + Return 1 if ``dec`` is ``NaN``, ``sNaN`` or ``Infinity``, 0 otherwise. + + Set TypeError and return -1 if ``dec`` is not a Decimal. It is guaranteed that + this is the only failure mode, so if ``dec`` has already been type-checked, no + errors can occur and the function can be treated as a simple predicate. + + +.. c:function:: int PyDec_IsNaN(const PyObject *dec) + + Return 1 if ``dec`` is ``NaN`` or ``sNaN``, 0 otherwise. + + Set TypeError and return -1 if ``dec`` is not a Decimal. It is guaranteed that + this is the only failure mode, so if ``dec`` has already been type-checked, no + errors can occur and the function can be treated as a simple predicate. + + +.. c:function:: int PyDec_IsInfinite(const PyObject *dec) + + Return 1 if ``dec`` is ``Infinity``, 0 otherwise. + + Set TypeError and return -1 if ``dec`` is not a Decimal. It is guaranteed that + this is the only failure mode, so if ``dec`` has already been type-checked, no + errors can occur and the function can be treated as a simple predicate. + + +.. c:function:: int64_t PyDec_GetDigits(const PyObject *dec) + + Return the number of digits in the coefficient. For ``Infinity``, the + number of digits is always zero. Typically, the same applies to ``NaN`` + and ``sNaN``, but both of these can have a payload that is equivalent to + a coefficient. Therefore, ``NaNs`` can have a nonzero return value. + + Set TypeError and return -1 if ``dec`` is not a Decimal. It is guaranteed that + this is the only failure mode, so if ``dec`` has already been type-checked, no + errors can occur and the function can be treated as a simple accessor. + + +Exact conversions between decimals and primitive C types +-------------------------------------------------------- + +This API supports conversions for decimals with a coefficient up to 38 digits. + +Data structures +~~~~~~~~~~~~~~~ + +The conversion functions use the following status codes and data structures: + +.. code-block:: + + /* status cases for getting a triple */ + enum mpd_triple_class { + MPD_TRIPLE_NORMAL, + MPD_TRIPLE_INF, + MPD_TRIPLE_QNAN, + MPD_TRIPLE_SNAN, + MPD_TRIPLE_ERROR, + }; + + typedef struct { + enum mpd_triple_class tag; + uint8_t sign; + uint64_t hi; + uint64_t lo; + int64_t exp; + } mpd_uint128_triple_t; + +The status cases are explained below. ``sign`` is 0 for positive and 1 for negative. +``((uint128_t)hi << 64) + lo`` is the coefficient, ``exp`` is the exponent. + +The data structure is called "triple" because the decimal triple (sign, coeff, exp) +is an established term and (``hi``, ``lo``) represents a single ``uint128_t`` coefficient. + + +Functions +~~~~~~~~~ + +.. c:function:: mpd_uint128_triple_t PyDec_AsUint128Triple(const PyObject *dec) + + Convert a decimal to a triple. As above, it is guaranteed that the only + Python failure mode is a TypeError, checks can be omitted if the type is + known. + + For simplicity, the usage of the function and all special cases are + explained in code form and comments: + +.. code-block:: + + triple = PyDec_AsUint128Triple(dec); + switch (triple.tag) { + case MPD_TRIPLE_QNAN: + /* + * Success: handle a quiet NaN. + * 1) triple.sign is 0 or 1. + * 2) triple.exp is always 0. + * 3) If triple.hi or triple.lo are nonzero, the NaN has a payload. + */ + break; + + case MPD_TRIPLE_SNAN: + /* + * Success: handle a signaling NaN. + * 1) triple.sign is 0 or 1. + * 2) triple.exp is always 0. + * 3) If triple.hi or triple.lo are nonzero, the sNaN has a payload. + */ + break; + + case MPD_TRIPLE_INF: + /* + * Success: handle Infinity. + * 1) triple.sign is 0 or 1. + * 2) triple.exp is always 0. + * 3) triple.hi and triple.lo are always zero. + */ + break; + + case MPD_TRIPLE_NORMAL: + /* Success: handle a finite value. */ + break; + + case MPD_TRIPLE_ERROR: + /* TypeError check: can be omitted if the type of dec is known. */ + if (PyErr_Occurred()) { + return NULL; + } + + /* Too large for conversion. PyDec_AsUint128Triple() does not set an + exception so applications can choose themselves. Typically this + would be a ValueError. */ + PyErr_SetString(PyExc_ValueError, + "value out of bounds for a uint128 triple"); + return NULL; + } + +.. c:function:: PyObject *PyDec_FromUint128Triple(const mpd_uint128_triple_t *triple) + + Create a decimal from a triple. The following rules must be observed for + initializing the triple: + + 1) ``triple.sign`` must always be 0 (for positive) or 1 (for negative). + + 2) ``MPD_TRIPLE_QNAN``: ``triple.exp`` must be 0. If ``triple.hi`` or ``triple.lo`` + are nonzero, create a ``NaN`` with a payload. + + 3) ``MPD_TRIPLE_SNAN``: ``triple.exp`` must be 0. If ``triple.hi`` or ``triple.lo`` + are nonzero, create an ``sNaN`` with a payload. + + 4) ``MPD_TRIPLE_INF``: ``triple.exp``, ``triple.hi`` and ``triple.lo`` must be zero. + + 5) ``MPD_TRIPLE_NORMAL``: ``MPD_MIN_ETINY + 38 < triple.exp < MPD_MAX_EMAX - 38``. + ``triple.hi`` and ``triple.lo`` can be chosen freely. + + 6) ``MPD_TRIPLE_ERROR``: It is always an error to set this tag. + + + If one of the above conditions is not met, the function returns ``NaN`` if + the ``InvalidOperation`` trap is not set in the thread local context. Otherwise, + it sets the ``InvalidOperation`` exception and returns NULL. + + Additionally, though extremely unlikely give the small allocation sizes, + the function can set ``MemoryError`` and return ``NULL``. + + +Advanced API +------------ + +This API enables the use of ``libmpdec`` functions. Since Python is compiled with +hidden symbols, the API requires an external libmpdec and the ``mpdecimal.h`` +header. + + +Functions +~~~~~~~~~ + +.. c:function:: PyObject *PyDec_Alloc(void) + + Return a new decimal that can be used in the ``result`` position of ``libmpdec`` + functions. + +.. c:function:: mpd_t *PyDec_Get(PyObject *v) + + Get a pointer to the internal ``mpd_t`` of the decimal. Decimals are immutable, + so this function must only be used on a new Decimal that has been created by + PyDec_Alloc(). + +.. c:function:: const mpd_t *PyDec_GetConst(const PyObject *v) + + Get a pointer to the constant internal ``mpd_t`` of the decimal. diff --git a/Include/pydecimal.h b/Include/pydecimal.h new file mode 100644 index 0000000000000..9b6440e1c2ab1 --- /dev/null +++ b/Include/pydecimal.h @@ -0,0 +1,180 @@ +/* + * Copyright (c) 2020 Stefan Krah. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + + +#ifndef CPYTHON_DECIMAL_H_ +#define CPYTHON_DECIMAL_H_ + + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/****************************************************************************/ +/* Libmpdec API */ +/****************************************************************************/ + +#ifndef LIBMPDEC_MPDECIMAL_H_ +struct mpd_t; /* ABI-stable in the libmpdec-2.x series */ + +/* status cases for getting a triple */ +enum mpd_triple_class { + MPD_TRIPLE_NORMAL, + MPD_TRIPLE_INF, + MPD_TRIPLE_QNAN, + MPD_TRIPLE_SNAN, + MPD_TRIPLE_ERROR, +}; + +typedef struct { + enum mpd_triple_class tag; + uint8_t sign; + uint64_t hi; + uint64_t lo; + int64_t exp; +} mpd_uint128_triple_t; +#endif + + +/****************************************************************************/ +/* Capsule API */ +/****************************************************************************/ + +/* Simple API */ +#define PyDec_TypeCheck_INDEX 0 +#define PyDec_TypeCheck_RETURN int +#define PyDec_TypeCheck_ARGS (const PyObject *) + +#define PyDec_IsSpecial_INDEX 1 +#define PyDec_IsSpecial_RETURN int +#define PyDec_IsSpecial_ARGS (const PyObject *) + +#define PyDec_IsNaN_INDEX 2 +#define PyDec_IsNaN_RETURN int +#define PyDec_IsNaN_ARGS (const PyObject *) + +#define PyDec_IsInfinite_INDEX 3 +#define PyDec_IsInfinite_RETURN int +#define PyDec_IsInfinite_ARGS (const PyObject *) + +#define PyDec_GetDigits_INDEX 4 +#define PyDec_GetDigits_RETURN int64_t +#define PyDec_GetDigits_ARGS (const PyObject *) + +#define PyDec_AsUint128Triple_INDEX 5 +#define PyDec_AsUint128Triple_RETURN mpd_uint128_triple_t +#define PyDec_AsUint128Triple_ARGS (const PyObject *) + +#define PyDec_FromUint128Triple_INDEX 6 +#define PyDec_FromUint128Triple_RETURN PyObject * +#define PyDec_FromUint128Triple_ARGS (const mpd_uint128_triple_t *triple) + +/* Advanced API */ +#define PyDec_Alloc_INDEX 7 +#define PyDec_Alloc_RETURN PyObject * +#define PyDec_Alloc_ARGS (void) + +#define PyDec_Get_INDEX 8 +#define PyDec_Get_RETURN mpd_t * +#define PyDec_Get_ARGS (PyObject *) + +#define PyDec_GetConst_INDEX 9 +#define PyDec_GetConst_RETURN const mpd_t * +#define PyDec_GetConst_ARGS (const PyObject *) + +#define CPYTHON_DECIMAL_MAX_API 10 + + +#ifdef CPYTHON_DECIMAL_MODULE +/* Simple API */ +static PyDec_TypeCheck_RETURN PyDec_TypeCheck PyDec_TypeCheck_ARGS; +static PyDec_IsSpecial_RETURN PyDec_IsSpecial PyDec_IsSpecial_ARGS; +static PyDec_IsNaN_RETURN PyDec_IsNaN PyDec_IsNaN_ARGS; +static PyDec_IsInfinite_RETURN PyDec_IsInfinite PyDec_IsInfinite_ARGS; +static PyDec_GetDigits_RETURN PyDec_GetDigits PyDec_GetDigits_ARGS; +static PyDec_AsUint128Triple_RETURN PyDec_AsUint128Triple PyDec_AsUint128Triple_ARGS; +static PyDec_FromUint128Triple_RETURN PyDec_FromUint128Triple PyDec_FromUint128Triple_ARGS; + +/* Advanced API */ +static PyDec_Alloc_RETURN PyDec_Alloc PyDec_Alloc_ARGS; +static PyDec_Get_RETURN PyDec_Get PyDec_Get_ARGS; +static PyDec_GetConst_RETURN PyDec_GetConst PyDec_GetConst_ARGS; +#else +static void **_decimal_api; + +/* Simple API */ +#define PyDec_TypeCheck \ + (*(PyDec_TypeCheck_RETURN (*)PyDec_TypeCheck_ARGS) _decimal_api[PyDec_TypeCheck_INDEX]) + +#define PyDec_IsSpecial \ + (*(PyDec_IsSpecial_RETURN (*)PyDec_IsSpecial_ARGS) _decimal_api[PyDec_IsSpecial_INDEX]) + +#define PyDec_IsNaN \ + (*(PyDec_IsNaN_RETURN (*)PyDec_IsNaN_ARGS) _decimal_api[PyDec_IsNaN_INDEX]) + +#define PyDec_IsInfinite \ + (*(PyDec_IsInfinite_RETURN (*)PyDec_IsInfinite_ARGS) _decimal_api[PyDec_IsInfinite_INDEX]) + +#define PyDec_GetDigits \ + (*(PyDec_GetDigits_RETURN (*)PyDec_GetDigits_ARGS) _decimal_api[PyDec_GetDigits_INDEX]) + +#define PyDec_AsUint128Triple \ + (*(PyDec_AsUint128Triple_RETURN (*)PyDec_AsUint128Triple_ARGS) _decimal_api[PyDec_AsUint128Triple_INDEX]) + +#define PyDec_FromUint128Triple \ + (*(PyDec_FromUint128Triple_RETURN (*)PyDec_FromUint128Triple_ARGS) _decimal_api[PyDec_FromUint128Triple_INDEX]) + +/* Advanced API */ +#define PyDec_Alloc \ + (*(PyDec_Alloc_RETURN (*)PyDec_Alloc_ARGS) _decimal_api[PyDec_Alloc_INDEX]) + +#define PyDec_Get \ + (*(PyDec_Get_RETURN (*)PyDec_Get_ARGS) _decimal_api[PyDec_Get_INDEX]) + +#define PyDec_GetConst \ + (*(PyDec_GetConst_RETURN (*)PyDec_GetConst_ARGS) _decimal_api[PyDec_GetConst_INDEX]) + + +static int +import_decimal(void) +{ + _decimal_api = (void **)PyCapsule_Import("_decimal._API", 0); + if (_decimal_api == NULL) { + return -1; + } + + return 0; +} +#endif + +#ifdef __cplusplus +} +#endif + +#endif /* CPYTHON_DECIMAL_H_ */ diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 5d0992a66e6ce..113b37ddaa9cd 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -43,6 +43,13 @@ import inspect import threading +from _testcapi import decimal_is_special +from _testcapi import decimal_is_nan +from _testcapi import decimal_is_infinite +from _testcapi import decimal_get_digits +from _testcapi import decimal_as_triple +from _testcapi import decimal_from_triple + C = import_fresh_module('decimal', fresh=['_decimal']) P = import_fresh_module('decimal', blocked=['_decimal']) @@ -4751,6 +4758,175 @@ def test_constants(self): self.assertEqual(C.DecTraps, C.DecErrors|C.DecOverflow|C.DecUnderflow) + def test_decimal_api_predicates(self): + # Capsule API + + d = C.Decimal("0") + self.assertFalse(decimal_is_special(d)) + self.assertFalse(decimal_is_nan(d)) + self.assertFalse(decimal_is_infinite(d)) + + d = C.Decimal("NaN") + self.assertTrue(decimal_is_special(d)) + self.assertTrue(decimal_is_nan(d)) + self.assertFalse(decimal_is_infinite(d)) + + d = C.Decimal("sNaN") + self.assertTrue(decimal_is_special(d)) + self.assertTrue(decimal_is_nan(d)) + self.assertFalse(decimal_is_infinite(d)) + + d = C.Decimal("inf") + self.assertTrue(decimal_is_special(d)) + self.assertFalse(decimal_is_nan(d)) + self.assertTrue(decimal_is_infinite(d)) + + def test_decimal_api_get_digits(self): + # Capsule API + + d = C.Decimal("0") + self.assertEqual(decimal_get_digits(d), 1) + + d = C.Decimal("1234567890") + self.assertEqual(decimal_get_digits(d), 10) + + d = C.Decimal("inf") + self.assertEqual(decimal_get_digits(d), 0) + + d = C.Decimal("NaN") + self.assertEqual(decimal_get_digits(d), 0) + + d = C.Decimal("sNaN") + self.assertEqual(decimal_get_digits(d), 0) + + d = C.Decimal("NaN1234567890") + self.assertEqual(decimal_get_digits(d), 10) + + d = C.Decimal("sNaN1234567890") + self.assertEqual(decimal_get_digits(d), 10) + + def test_decimal_api_triple(self): + # Capsule API + + def as_triple(d): + """Convert a decimal to a decimal triple with a split uint128_t + coefficient: + + (sign, hi, lo, exp) + + It is called 'triple' because (hi, lo) are regarded as a single + uint128_t that is split because not all compilers support uint128_t. + """ + sign, digits, exp = d.as_tuple() + + s = "".join(str(d) for d in digits) + coeff = int(s) if s else 0 + + if coeff < 0 or coeff >= 2**128: + raise ValueError("value out of bounds for a uint128 triple"); + + hi, lo = divmod(coeff, 2**64) + return (sign, hi, lo, exp) + + def from_triple(triple): + """Convert a decimal triple with a split uint128_t coefficient to a string. + """ + sign, hi, lo, exp = triple + coeff = hi * 2**64 + lo + + if coeff < 0 or coeff >= 2**128: + raise ValueError("value out of bounds for a uint128 triple"); + + digits = tuple(int(c) for c in str(coeff)) + + return P.Decimal((sign, digits, exp)) + + signs = ["", "-"] + + coefficients = [ + "000000000000000000000000000000000000000", + + "299999999999999999999999999999999999999", + "299999999999999999990000000000000000000", + "200000000000000000009999999999999999999", + "000000000000000000009999999999999999999", + + "299999999999999999999999999999000000000", + "299999999999999999999000000000999999999", + "299999999999000000000999999999999999999", + "299000000000999999999999999999999999999", + "000999999999999999999999999999999999999", + + "300000000000000000000000000000000000000", + "310000000000000000001000000000000000000", + "310000000000000000000000000000000000000", + "300000000000000000001000000000000000000", + + "340100000000100000000100000000100000000", + "340100000000100000000100000000000000000", + "340100000000100000000000000000100000000", + "340100000000000000000100000000100000000", + "340000000000100000000100000000100000000", + + "340282366920938463463374607431768211455", + ] + + exponents = [ + "E+0", "E+1", "E-1", + "E+%s" % str(C.MAX_EMAX-38), + "E-%s" % str(C.MIN_ETINY+38), + ] + + for sign in signs: + for coeff in coefficients: + for exp in exponents: + s = sign + coeff + exp + + ctriple = decimal_as_triple(C.Decimal(s)) + ptriple = as_triple(P.Decimal(s)) + self.assertEqual(ctriple, ptriple) + + c = decimal_from_triple(ctriple) + p = decimal_from_triple(ptriple) + self.assertEqual(str(c), str(p)) + + for s in ["NaN", "-NaN", "sNaN", "-sNaN", "NaN123", "sNaN123", "inf", "-inf"]: + ctriple = decimal_as_triple(C.Decimal(s)) + ptriple = as_triple(P.Decimal(s)) + self.assertEqual(ctriple, ptriple) + + c = decimal_from_triple(ctriple) + p = decimal_from_triple(ptriple) + self.assertEqual(str(c), str(p)) + + def test_decimal_api_errors(self): + # Capsule API + + self.assertRaises(TypeError, decimal_as_triple, "X") + self.assertRaises(ValueError, decimal_as_triple, C.Decimal(2**128)) + self.assertRaises(ValueError, decimal_as_triple, C.Decimal(-2**128)) + + self.assertRaises(TypeError, decimal_from_triple, "X") + self.assertRaises(ValueError, decimal_from_triple, ()) + self.assertRaises(ValueError, decimal_from_triple, (1, 2, 3, 4, 5)) + self.assertRaises(ValueError, decimal_from_triple, (2**8, 0, 0, 0)) + self.assertRaises(OverflowError, decimal_from_triple, (0, 2**64, 0, 0)) + self.assertRaises(OverflowError, decimal_from_triple, (0, 0, 2**64, 0)) + self.assertRaises(OverflowError, decimal_from_triple, (0, 0, 0, 2**63)) + self.assertRaises(OverflowError, decimal_from_triple, (0, 0, 0, -2**63-1)) + self.assertRaises(ValueError, decimal_from_triple, (0, 0, 0, "X")) + self.assertRaises(TypeError, decimal_from_triple, (0, 0, 0, ())) + + with C.localcontext(C.Context()): + self.assertRaises(C.InvalidOperation, decimal_from_triple, (2, 0, 0, 0)) + self.assertRaises(C.InvalidOperation, decimal_from_triple, (0, 0, 0, 2**63-1)) + self.assertRaises(C.InvalidOperation, decimal_from_triple, (0, 0, 0, -2**63)) + + self.assertRaises(TypeError, decimal_is_special, "X") + self.assertRaises(TypeError, decimal_is_nan, "X") + self.assertRaises(TypeError, decimal_is_infinite, "X") + self.assertRaises(TypeError, decimal_get_digits, "X") + class CWhitebox(unittest.TestCase): """Whitebox testing for _decimal""" diff --git a/Misc/NEWS.d/next/C API/2020-08-10-16-05-08.bpo-41324.waZD35.rst b/Misc/NEWS.d/next/C API/2020-08-10-16-05-08.bpo-41324.waZD35.rst new file mode 100644 index 0000000000000..e09332ab11e1d --- /dev/null +++ b/Misc/NEWS.d/next/C API/2020-08-10-16-05-08.bpo-41324.waZD35.rst @@ -0,0 +1,3 @@ +Add a minimal decimal capsule API. The API supports fast conversions +between Decimals up to 38 digits and their triple representation as a C +struct. diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index fb4e020f1260e..e7c44acba02fc 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2020 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -33,6 +33,8 @@ #include +#define CPYTHON_DECIMAL_MODULE +#include "pydecimal.h" #include "docstrings.h" @@ -5555,6 +5557,160 @@ static PyTypeObject PyDecContext_Type = }; +/****************************************************************************/ +/* C-API */ +/****************************************************************************/ + +static void *_decimal_api[CPYTHON_DECIMAL_MAX_API]; + +/* Simple API */ +static int +PyDec_TypeCheck(const PyObject *v) +{ + return PyDec_Check(v); +} + +static int +PyDec_IsSpecial(const PyObject *v) +{ + if (!PyDec_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "PyDec_IsSpecial: argument must be a Decimal"); + return -1; + } + + return mpd_isspecial(MPD(v)); +} + +static int +PyDec_IsNaN(const PyObject *v) +{ + if (!PyDec_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "PyDec_IsNaN: argument must be a Decimal"); + return -1; + } + + return mpd_isnan(MPD(v)); +} + +static int +PyDec_IsInfinite(const PyObject *v) +{ + if (!PyDec_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "PyDec_IsInfinite: argument must be a Decimal"); + return -1; + } + + return mpd_isinfinite(MPD(v)); +} + +static int64_t +PyDec_GetDigits(const PyObject *v) +{ + if (!PyDec_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "PyDec_GetDigits: argument must be a Decimal"); + return -1; + } + + return MPD(v)->digits; +} + +static mpd_uint128_triple_t +PyDec_AsUint128Triple(const PyObject *v) +{ + if (!PyDec_Check(v)) { + mpd_uint128_triple_t triple = { MPD_TRIPLE_ERROR, 0, 0, 0, 0 }; + PyErr_SetString(PyExc_TypeError, + "PyDec_AsUint128Triple: argument must be a Decimal"); + return triple; + } + + return mpd_as_uint128_triple(MPD(v)); +} + +static PyObject * +PyDec_FromUint128Triple(const mpd_uint128_triple_t *triple) +{ + PyObject *context; + PyObject *result; + uint32_t status = 0; + + CURRENT_CONTEXT(context); + + result = dec_alloc(); + if (result == NULL) { + return NULL; + } + + if (mpd_from_uint128_triple(MPD(result), triple, &status) < 0) { + if (dec_addstatus(context, status)) { + Py_DECREF(result); + return NULL; + } + } + + return result; +} + +/* Advanced API */ +static PyObject * +PyDec_Alloc(void) +{ + return dec_alloc(); +} + +static mpd_t * +PyDec_Get(PyObject *v) +{ + if (!PyDec_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "PyDec_Get: argument must be a Decimal"); + return NULL; + } + + return MPD(v); +} + +static const mpd_t * +PyDec_GetConst(const PyObject *v) +{ + if (!PyDec_Check(v)) { + PyErr_SetString(PyExc_TypeError, + "PyDec_GetConst: argument must be a Decimal"); + return NULL; + } + + return MPD(v); +} + +static PyObject * +init_api(void) +{ + /* Simple API */ + _decimal_api[PyDec_TypeCheck_INDEX] = (void *)PyDec_TypeCheck; + _decimal_api[PyDec_IsSpecial_INDEX] = (void *)PyDec_IsSpecial; + _decimal_api[PyDec_IsNaN_INDEX] = (void *)PyDec_IsNaN; + _decimal_api[PyDec_IsInfinite_INDEX] = (void *)PyDec_IsInfinite; + _decimal_api[PyDec_GetDigits_INDEX] = (void *)PyDec_GetDigits; + _decimal_api[PyDec_AsUint128Triple_INDEX] = (void *)PyDec_AsUint128Triple; + _decimal_api[PyDec_FromUint128Triple_INDEX] = (void *)PyDec_FromUint128Triple; + + /* Advanced API */ + _decimal_api[PyDec_Alloc_INDEX] = (void *)PyDec_Alloc; + _decimal_api[PyDec_Get_INDEX] = (void *)PyDec_Get; + _decimal_api[PyDec_GetConst_INDEX] = (void *)PyDec_GetConst; + + return PyCapsule_New(_decimal_api, "_decimal._API", NULL); +} + + +/****************************************************************************/ +/* Module */ +/****************************************************************************/ + static PyMethodDef _decimal_methods [] = { { "getcontext", (PyCFunction)PyDec_GetCurrentContext, METH_NOARGS, doc_getcontext}, @@ -5665,17 +5821,27 @@ PyInit__decimal(void) DecCondMap *cm; struct ssize_constmap *ssize_cm; struct int_constmap *int_cm; + static PyObject *capsule = NULL; + static int initialized = 0; int i; /* Init libmpdec */ - mpd_traphandler = dec_traphandler; - mpd_mallocfunc = PyMem_Malloc; - mpd_reallocfunc = PyMem_Realloc; - mpd_callocfunc = mpd_callocfunc_em; - mpd_free = PyMem_Free; - mpd_setminalloc(_Py_DEC_MINALLOC); + if (!initialized) { + mpd_traphandler = dec_traphandler; + mpd_mallocfunc = PyMem_Malloc; + mpd_reallocfunc = PyMem_Realloc; + mpd_callocfunc = mpd_callocfunc_em; + mpd_free = PyMem_Free; + mpd_setminalloc(_Py_DEC_MINALLOC); + + capsule = init_api(); + if (capsule == NULL) { + return NULL; + } + initialized = 1; + } /* Init external C-API functions */ _py_long_multiply = PyLong_Type.tp_as_number->nb_multiply; @@ -5900,6 +6066,11 @@ PyInit__decimal(void) CHECK_INT(PyModule_AddStringConstant(m, "__version__", "1.70")); CHECK_INT(PyModule_AddStringConstant(m, "__libmpdec_version__", mpd_version())); + /* Add capsule API */ + Py_INCREF(capsule); + if (PyModule_AddObject(m, "_API", capsule) < 0) { + goto error; + } return m; diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index 5d9179e61689d..15f104dc463cb 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -49,6 +49,9 @@ from formathelper import rand_format, rand_locale from _pydecimal import _dec_from_triple +from _testcapi import decimal_as_triple +from _testcapi import decimal_from_triple + C = import_fresh_module('decimal', fresh=['_decimal']) P = import_fresh_module('decimal', blocked=['_decimal']) EXIT_STATUS = 0 @@ -153,6 +156,45 @@ TernaryRestricted = ['__pow__', 'context.power'] +# ====================================================================== +# Triple tests +# ====================================================================== + +def c_as_triple(dec): + sign, hi, lo, exp = decimal_as_triple(dec) + + coeff = hi * 2**64 + lo + return (sign, coeff, exp) + +def c_from_triple(triple): + sign, coeff, exp = triple + + hi = coeff // 2**64 + lo = coeff % 2**64 + return decimal_from_triple((sign, hi, lo, exp)) + +def p_as_triple(dec): + sign, digits, exp = dec.as_tuple() + + s = "".join(str(d) for d in digits) + coeff = int(s) if s else 0 + + if coeff < 0 or coeff >= 2**128: + raise ValueError("value out of bounds for a uint128 triple"); + + return (sign, coeff, exp) + +def p_from_triple(triple): + sign, coeff, exp = triple + + if coeff < 0 or coeff >= 2**128: + raise ValueError("value out of bounds for a uint128 triple"); + + digits = tuple(int(c) for c in str(coeff)) + + return P.Decimal((sign, digits, exp)) + + # ====================================================================== # Unified Context # ====================================================================== @@ -846,12 +888,44 @@ def verify(t, stat): t.presults.append(str(t.rp.imag)) t.presults.append(str(t.rp.real)) + ctriple = None + if t.funcname not in ['__radd__', '__rmul__']: # see skip handler + try: + ctriple = c_as_triple(t.rc) + except ValueError: + try: + ptriple = p_as_triple(t.rp) + except ValueError: + pass + else: + raise RuntimeError("ValueError not raised") + else: + cres = c_from_triple(ctriple) + t.cresults.append(ctriple) + t.cresults.append(str(cres)) + + ptriple = p_as_triple(t.rp) + pres = p_from_triple(ptriple) + t.presults.append(ptriple) + t.presults.append(str(pres)) + if t.with_maxcontext and isinstance(t.rmax, C.Decimal): t.maxresults.append(t.rmax.to_eng_string()) t.maxresults.append(t.rmax.as_tuple()) t.maxresults.append(str(t.rmax.imag)) t.maxresults.append(str(t.rmax.real)) + if ctriple is not None: + # NaN payloads etc. depend on precision and clamp. + if all_nan(t.rc) and all_nan(t.rmax): + t.maxresults.append(ctriple) + t.maxresults.append(str(cres)) + else: + maxtriple = c_as_triple(t.rmax) + maxres = c_from_triple(maxtriple) + t.maxresults.append(maxtriple) + t.maxresults.append(str(maxres)) + nc = t.rc.number_class().lstrip('+-s') stat[nc] += 1 else: diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index fca94a83a5d04..593034ef65e2c 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -19,6 +19,7 @@ #include "Python.h" #include "datetime.h" +#include "pydecimal.h" #include "marshal.h" #include "structmember.h" // PyMemberDef #include @@ -2705,6 +2706,252 @@ test_PyDateTime_DELTA_GET(PyObject *self, PyObject *obj) return Py_BuildValue("(lll)", days, seconds, microseconds); } +/* Test decimal API */ +static int decimal_initialized = 0; +static PyObject * +decimal_is_special(PyObject *module, PyObject *dec) +{ + int is_special; + + (void)module; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + is_special = PyDec_IsSpecial(dec); + if (is_special < 0) { + return NULL; + } + + return PyBool_FromLong(is_special); +} + +static PyObject * +decimal_is_nan(PyObject *module, PyObject *dec) +{ + int is_nan; + + (void)module; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + is_nan = PyDec_IsNaN(dec); + if (is_nan < 0) { + return NULL; + } + + return PyBool_FromLong(is_nan); +} + +static PyObject * +decimal_is_infinite(PyObject *module, PyObject *dec) +{ + int is_infinite; + + (void)module; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + is_infinite = PyDec_IsInfinite(dec); + if (is_infinite < 0) { + return NULL; + } + + return PyBool_FromLong(is_infinite); +} + +static PyObject * +decimal_get_digits(PyObject *module, PyObject *dec) +{ + int64_t digits; + + (void)module; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + digits = PyDec_GetDigits(dec); + if (digits < 0) { + return NULL; + } + + return PyLong_FromLongLong(digits); +} + +static PyObject * +decimal_as_triple(PyObject *module, PyObject *dec) +{ + PyObject *tuple = NULL; + PyObject *sign, *hi, *lo; + mpd_uint128_triple_t triple; + + (void)module; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + triple = PyDec_AsUint128Triple(dec); + if (triple.tag == MPD_TRIPLE_ERROR && PyErr_Occurred()) { + return NULL; + } + + sign = PyLong_FromUnsignedLong(triple.sign); + if (sign == NULL) { + return NULL; + } + + hi = PyLong_FromUnsignedLongLong(triple.hi); + if (hi == NULL) { + Py_DECREF(sign); + return NULL; + } + + lo = PyLong_FromUnsignedLongLong(triple.lo); + if (lo == NULL) { + Py_DECREF(hi); + Py_DECREF(sign); + return NULL; + } + + switch (triple.tag) { + case MPD_TRIPLE_QNAN: + assert(triple.exp == 0); + tuple = Py_BuildValue("(OOOs)", sign, hi, lo, "n"); + break; + + case MPD_TRIPLE_SNAN: + assert(triple.exp == 0); + tuple = Py_BuildValue("(OOOs)", sign, hi, lo, "N"); + break; + + case MPD_TRIPLE_INF: + assert(triple.hi == 0); + assert(triple.lo == 0); + assert(triple.exp == 0); + tuple = Py_BuildValue("(OOOs)", sign, hi, lo, "F"); + break; + + case MPD_TRIPLE_NORMAL: + tuple = Py_BuildValue("(OOOL)", sign, hi, lo, triple.exp); + break; + + case MPD_TRIPLE_ERROR: + PyErr_SetString(PyExc_ValueError, + "value out of bounds for a uint128 triple"); + break; + + default: + PyErr_SetString(PyExc_RuntimeError, + "decimal_as_triple: internal error: unexpected tag"); + break; + } + + Py_DECREF(lo); + Py_DECREF(hi); + Py_DECREF(sign); + + return tuple; +} + +static PyObject * +decimal_from_triple(PyObject *module, PyObject *tuple) +{ + mpd_uint128_triple_t triple = { MPD_TRIPLE_ERROR, 0, 0, 0, 0 }; + PyObject *exp; + unsigned long sign; + + (void)module; + if (!decimal_initialized) { + if (import_decimal() < 0) { + return NULL; + } + + decimal_initialized = 1; + } + + if (!PyTuple_Check(tuple)) { + PyErr_SetString(PyExc_TypeError, "argument must be a tuple"); + return NULL; + } + + if (PyTuple_GET_SIZE(tuple) != 4) { + PyErr_SetString(PyExc_ValueError, "tuple size must be 4"); + return NULL; + } + + sign = PyLong_AsUnsignedLong(PyTuple_GET_ITEM(tuple, 0)); + if (sign == (unsigned long)-1 && PyErr_Occurred()) { + return NULL; + } + if (sign > UINT8_MAX) { + PyErr_SetString(PyExc_ValueError, "sign must be 0 or 1"); + return NULL; + } + triple.sign = (uint8_t)sign; + + triple.hi = PyLong_AsUnsignedLongLong(PyTuple_GET_ITEM(tuple, 1)); + if (triple.hi == (unsigned long long)-1 && PyErr_Occurred()) { + return NULL; + } + + triple.lo = PyLong_AsUnsignedLongLong(PyTuple_GET_ITEM(tuple, 2)); + if (triple.lo == (unsigned long long)-1 && PyErr_Occurred()) { + return NULL; + } + + exp = PyTuple_GET_ITEM(tuple, 3); + if (PyLong_Check(exp)) { + triple.tag = MPD_TRIPLE_NORMAL; + triple.exp = PyLong_AsLongLong(exp); + if (triple.exp == -1 && PyErr_Occurred()) { + return NULL; + } + } + else if (PyUnicode_Check(exp)) { + if (PyUnicode_CompareWithASCIIString(exp, "F") == 0) { + triple.tag = MPD_TRIPLE_INF; + } + else if (PyUnicode_CompareWithASCIIString(exp, "n") == 0) { + triple.tag = MPD_TRIPLE_QNAN; + } + else if (PyUnicode_CompareWithASCIIString(exp, "N") == 0) { + triple.tag = MPD_TRIPLE_SNAN; + } + else { + PyErr_SetString(PyExc_ValueError, "not a valid exponent"); + return NULL; + } + } + else { + PyErr_SetString(PyExc_TypeError, "exponent must be int or string"); + return NULL; + } + + return PyDec_FromUint128Triple(&triple); +} + /* test_thread_state spawns a thread of its own, and that thread releases * `thread_done` when it's finished. The driver code has to know when the * thread finishes, because the thread uses a PyObject (the callable) that @@ -5314,6 +5561,12 @@ static PyMethodDef TestMethods[] = { {"PyDateTime_DATE_GET", test_PyDateTime_DATE_GET, METH_O}, {"PyDateTime_TIME_GET", test_PyDateTime_TIME_GET, METH_O}, {"PyDateTime_DELTA_GET", test_PyDateTime_DELTA_GET, METH_O}, + {"decimal_is_special", decimal_is_special, METH_O}, + {"decimal_is_nan", decimal_is_nan, METH_O}, + {"decimal_is_infinite", decimal_is_infinite, METH_O}, + {"decimal_get_digits", decimal_get_digits, METH_O}, + {"decimal_as_triple", decimal_as_triple, METH_O}, + {"decimal_from_triple", decimal_from_triple, METH_O}, {"test_list_api", test_list_api, METH_NOARGS}, {"test_dict_iteration", test_dict_iteration, METH_NOARGS}, {"dict_getitem_knownhash", dict_getitem_knownhash, METH_VARARGS}, From webhook-mailer at python.org Mon Aug 10 12:37:08 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 10 Aug 2020 16:37:08 -0000 Subject: [Python-checkins] bpo-40548: Fix "Check for source changes (pull_request)" GH Action job (GH-21806) Message-ID: https://github.com/python/cpython/commit/eaa551702d80fd67219c48ee6a13ffb571ca360b commit: eaa551702d80fd67219c48ee6a13ffb571ca360b branch: master author: Victor Stinner committer: GitHub date: 2020-08-10T18:36:59+02:00 summary: bpo-40548: Fix "Check for source changes (pull_request)" GH Action job (GH-21806) On Git 2.28, "git diff master..." (3 dots) no longer works when "fetch --depth=1" is used, whereas it works on Git 2.26. Replace "..." (3 dots) with ".." (2 dots) in the "git diff" command computing the list of modified files between the base branch and the PR branch. files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 5649a6670e75f..df68fe271de7a 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -32,7 +32,20 @@ jobs: echo '::set-output name=run_tests::true' else git fetch origin $GITHUB_BASE_REF --depth=1 - git diff --name-only origin/$GITHUB_BASE_REF... | grep -qvE '(\.rst$|^Doc|^Misc)' && echo '::set-output name=run_tests::true' || true + # git diff "origin/$GITHUB_BASE_REF..." (3 dots) may be more + # reliable than git diff "origin/$GITHUB_BASE_REF.." (2 dots), + # but it requires to download more commits (this job uses + # "git fetch --depth=1"). + # + # git diff "origin/$GITHUB_BASE_REF..." (3 dots) works with Git + # 2.26, but Git 2.28 is stricter and fails with "no merge base". + # + # git diff "origin/$GITHUB_BASE_REF.." (2 dots) should be enough on + # GitHub, since GitHub starts by merging origin/$GITHUB_BASE_REF + # into the PR branch anyway. + # + # https://github.com/python/core-workflow/issues/373 + git diff --name-only origin/$GITHUB_BASE_REF.. | grep -qvE '(\.rst$|^Doc|^Misc)' && echo '::set-output name=run_tests::true' || true fi build_win32: name: 'Windows (x86)' From webhook-mailer at python.org Mon Aug 10 13:06:20 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 10 Aug 2020 17:06:20 -0000 Subject: [Python-checkins] bpo-40548: Fix "Check for source changes (pull_request)" GH Action job (GH-21806) Message-ID: https://github.com/python/cpython/commit/09d82609be3d4903104610ed918caeefb953f79a commit: 09d82609be3d4903104610ed918caeefb953f79a branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-10T10:06:11-07:00 summary: bpo-40548: Fix "Check for source changes (pull_request)" GH Action job (GH-21806) On Git 2.28, "git diff master..." (3 dots) no longer works when "fetch --depth=1" is used, whereas it works on Git 2.26. Replace "..." (3 dots) with ".." (2 dots) in the "git diff" command computing the list of modified files between the base branch and the PR branch. (cherry picked from commit eaa551702d80fd67219c48ee6a13ffb571ca360b) Co-authored-by: Victor Stinner files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index cf86505980244..1d0cf0b5ec352 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -25,7 +25,20 @@ jobs: echo '::set-output name=run_tests::true' else git fetch origin $GITHUB_BASE_REF --depth=1 - git diff --name-only origin/$GITHUB_BASE_REF... | grep -qvE '(\.rst$|^Doc|^Misc)' && echo '::set-output name=run_tests::true' || true + # git diff "origin/$GITHUB_BASE_REF..." (3 dots) may be more + # reliable than git diff "origin/$GITHUB_BASE_REF.." (2 dots), + # but it requires to download more commits (this job uses + # "git fetch --depth=1"). + # + # git diff "origin/$GITHUB_BASE_REF..." (3 dots) works with Git + # 2.26, but Git 2.28 is stricter and fails with "no merge base". + # + # git diff "origin/$GITHUB_BASE_REF.." (2 dots) should be enough on + # GitHub, since GitHub starts by merging origin/$GITHUB_BASE_REF + # into the PR branch anyway. + # + # https://github.com/python/core-workflow/issues/373 + git diff --name-only origin/$GITHUB_BASE_REF.. | grep -qvE '(\.rst$|^Doc|^Misc)' && echo '::set-output name=run_tests::true' || true fi build_win32: name: 'Windows (x86)' From webhook-mailer at python.org Mon Aug 10 15:54:55 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Mon, 10 Aug 2020 19:54:55 -0000 Subject: [Python-checkins] Replace import_fresh_module in decimal test files (GH-21815) Message-ID: https://github.com/python/cpython/commit/85fdafa6ea933aa0623f5d8297ab0f4b20866234 commit: 85fdafa6ea933aa0623f5d8297ab0f4b20866234 branch: master author: Stefan Krah committer: GitHub date: 2020-08-10T21:54:50+02:00 summary: Replace import_fresh_module in decimal test files (GH-21815) files: M Modules/_decimal/tests/bench.py M Modules/_decimal/tests/deccheck.py M Modules/_decimal/tests/formathelper.py diff --git a/Modules/_decimal/tests/bench.py b/Modules/_decimal/tests/bench.py index 3726db194e032..88fd7b5ae0be3 100644 --- a/Modules/_decimal/tests/bench.py +++ b/Modules/_decimal/tests/bench.py @@ -7,13 +7,10 @@ import time -try: - from test.support import import_fresh_module -except ImportError: - from test.test_support import import_fresh_module -C = import_fresh_module('decimal', fresh=['_decimal']) -P = import_fresh_module('decimal', blocked=['_decimal']) +import _decimal as C +import _pydecimal as P + # # NOTE: This is the pi function from the decimal documentation, modified diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index 15f104dc463cb..0b2a1c49336ef 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -43,7 +43,6 @@ from queue import Queue, Empty from threading import Thread, Event, Lock -from test.support import import_fresh_module from randdec import randfloat, all_unary, all_binary, all_ternary from randdec import unary_optarg, binary_optarg, ternary_optarg from formathelper import rand_format, rand_locale @@ -52,8 +51,10 @@ from _testcapi import decimal_as_triple from _testcapi import decimal_from_triple -C = import_fresh_module('decimal', fresh=['_decimal']) -P = import_fresh_module('decimal', blocked=['_decimal']) +import _decimal as C +import _pydecimal as P + + EXIT_STATUS = 0 diff --git a/Modules/_decimal/tests/formathelper.py b/Modules/_decimal/tests/formathelper.py index 19b2aad4a503b..482e02a25c2a8 100644 --- a/Modules/_decimal/tests/formathelper.py +++ b/Modules/_decimal/tests/formathelper.py @@ -31,11 +31,10 @@ import os, sys, locale, random import platform, subprocess -from test.support import import_fresh_module from distutils.spawn import find_executable -C = import_fresh_module('decimal', fresh=['_decimal']) -P = import_fresh_module('decimal', blocked=['_decimal']) +import _decimal as C +import _pydecimal as P windows_lang_strings = [ From webhook-mailer at python.org Mon Aug 10 16:39:56 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Mon, 10 Aug 2020 20:39:56 -0000 Subject: [Python-checkins] MSVC: The ARM command line should not define MASM. (#21817) Message-ID: https://github.com/python/cpython/commit/a02efe4d27c2aaa4af0cfaf87efa2146b677b17d commit: a02efe4d27c2aaa4af0cfaf87efa2146b677b17d branch: master author: Stefan Krah committer: GitHub date: 2020-08-10T22:39:46+02:00 summary: MSVC: The ARM command line should not define MASM. (#21817) files: M PCbuild/_decimal.vcxproj diff --git a/PCbuild/_decimal.vcxproj b/PCbuild/_decimal.vcxproj index 4c71cdb6d1d77..0916f1a2d3788 100644 --- a/PCbuild/_decimal.vcxproj +++ b/PCbuild/_decimal.vcxproj @@ -93,11 +93,11 @@ - _CRT_SECURE_NO_WARNINGS;MASM;%(PreprocessorDefinitions) - CONFIG_32;PPRO;%(PreprocessorDefinitions) + _CRT_SECURE_NO_WARNINGS;%(PreprocessorDefinitions) + CONFIG_32;PPRO;MASM;%(PreprocessorDefinitions) CONFIG_32;ANSI;%(PreprocessorDefinitions) CONFIG_64;ANSI;%(PreprocessorDefinitions) - CONFIG_64;%(PreprocessorDefinitions) + CONFIG_64;MASM;%(PreprocessorDefinitions) ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) From webhook-mailer at python.org Mon Aug 10 17:24:12 2020 From: webhook-mailer at python.org (Hai Shi) Date: Mon, 10 Aug 2020 21:24:12 -0000 Subject: [Python-checkins] bpo-40275: Fix failed test cases by using test helpers (GH-21811) Message-ID: https://github.com/python/cpython/commit/490c5426b1b23831d83d0c6b269858fb98450889 commit: 490c5426b1b23831d83d0c6b269858fb98450889 branch: master author: Hai Shi committer: GitHub date: 2020-08-10T23:24:02+02:00 summary: bpo-40275: Fix failed test cases by using test helpers (GH-21811) files: M Lib/test/test__osx_support.py M Lib/test/test_importlib/extension/test_case_sensitivity.py M Lib/test/test_importlib/source/test_case_sensitivity.py M Lib/test/test_selectors.py diff --git a/Lib/test/test__osx_support.py b/Lib/test/test__osx_support.py index a3f41d2c5bd07..907ae27d529b5 100644 --- a/Lib/test/test__osx_support.py +++ b/Lib/test/test__osx_support.py @@ -8,7 +8,6 @@ import sys import unittest -import test.support from test.support import os_helper import _osx_support @@ -20,7 +19,7 @@ def setUp(self): self.maxDiff = None self.prog_name = 'bogus_program_xxxx' self.temp_path_dir = os.path.abspath(os.getcwd()) - self.env = test.support.EnvironmentVarGuard() + self.env = os_helper.EnvironmentVarGuard() self.addCleanup(self.env.__exit__) for cv in ('CFLAGS', 'LDFLAGS', 'CPPFLAGS', 'BASECFLAGS', 'BLDSHARED', 'LDSHARED', 'CC', diff --git a/Lib/test/test_importlib/extension/test_case_sensitivity.py b/Lib/test/test_importlib/extension/test_case_sensitivity.py index 3a857847381a9..4d76fa014cd73 100644 --- a/Lib/test/test_importlib/extension/test_case_sensitivity.py +++ b/Lib/test/test_importlib/extension/test_case_sensitivity.py @@ -1,5 +1,5 @@ from importlib import _bootstrap_external -from test import support +from test.support import os_helper import unittest import sys from .. import util @@ -23,7 +23,7 @@ def find_module(self): @unittest.skipIf(sys.flags.ignore_environment, 'ignore_environment flag was set') def test_case_sensitive(self): - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.unset('PYTHONCASEOK') self.caseok_env_changed(should_exist=False) loader = self.find_module() @@ -31,7 +31,7 @@ def test_case_sensitive(self): @unittest.skipIf(sys.flags.ignore_environment, 'ignore_environment flag was set') def test_case_insensitivity(self): - with support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.set('PYTHONCASEOK', '1') self.caseok_env_changed(should_exist=True) loader = self.find_module() diff --git a/Lib/test/test_importlib/source/test_case_sensitivity.py b/Lib/test/test_importlib/source/test_case_sensitivity.py index ad1cfdb7e5cc1..77c06a75fc77b 100644 --- a/Lib/test/test_importlib/source/test_case_sensitivity.py +++ b/Lib/test/test_importlib/source/test_case_sensitivity.py @@ -7,7 +7,7 @@ machinery = util.import_importlib('importlib.machinery') import os -from test import support as test_support +from test.support import os_helper import unittest @@ -42,7 +42,7 @@ def sensitivity_test(self): @unittest.skipIf(sys.flags.ignore_environment, 'ignore_environment flag was set') def test_sensitive(self): - with test_support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.unset('PYTHONCASEOK') self.caseok_env_changed(should_exist=False) sensitive, insensitive = self.sensitivity_test() @@ -52,7 +52,7 @@ def test_sensitive(self): @unittest.skipIf(sys.flags.ignore_environment, 'ignore_environment flag was set') def test_insensitive(self): - with test_support.EnvironmentVarGuard() as env: + with os_helper.EnvironmentVarGuard() as env: env.set('PYTHONCASEOK', '1') self.caseok_env_changed(should_exist=True) sensitive, insensitive = self.sensitivity_test() diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py index 2274c39a79a53..851b1fc1b9e0c 100644 --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -6,6 +6,7 @@ import socket import sys from test import support +from test.support import os_helper from test.support import socket_helper from time import sleep import unittest @@ -536,7 +537,7 @@ def test_register_bad_fd(self): # a file descriptor that's been closed should raise an OSError # with EBADF s = self.SELECTOR() - bad_f = support.make_bad_fd() + bad_f = os_helper.make_bad_fd() with self.assertRaises(OSError) as cm: s.register(bad_f, selectors.EVENT_READ) self.assertEqual(cm.exception.errno, errno.EBADF) From webhook-mailer at python.org Tue Aug 11 06:32:51 2020 From: webhook-mailer at python.org (Mohamed Koubaa) Date: Tue, 11 Aug 2020 10:32:51 -0000 Subject: [Python-checkins] bpo-1635741: Port multiprocessing ext to multiphase init (GH-21378) Message-ID: https://github.com/python/cpython/commit/1d541c25c8019f7a0b80b3e1b437abe171e40b65 commit: 1d541c25c8019f7a0b80b3e1b437abe171e40b65 branch: master author: Mohamed Koubaa committer: GitHub date: 2020-08-11T12:32:35+02:00 summary: bpo-1635741: Port multiprocessing ext to multiphase init (GH-21378) Port the _multiprocessing extension module to multiphase initialization (PEP 489). files: A Misc/NEWS.d/next/Core and Builtins/2020-07-07-16-10-52.bpo-1635741.zU-H_n.rst M Modules/_multiprocessing/multiprocessing.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-07-07-16-10-52.bpo-1635741.zU-H_n.rst b/Misc/NEWS.d/next/Core and Builtins/2020-07-07-16-10-52.bpo-1635741.zU-H_n.rst new file mode 100644 index 0000000000000..52e184dc31707 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-07-07-16-10-52.bpo-1635741.zU-H_n.rst @@ -0,0 +1 @@ +Port :mod:`multiprocessing` to multi-phase initialization diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c index 77e6c854068c0..25b8dc3967a4f 100644 --- a/Modules/_multiprocessing/multiprocessing.c +++ b/Modules/_multiprocessing/multiprocessing.c @@ -183,35 +183,17 @@ static PyMethodDef module_methods[] = { * Initialize */ -static struct PyModuleDef multiprocessing_module = { - PyModuleDef_HEAD_INIT, - "_multiprocessing", - NULL, - -1, - module_methods, - NULL, - NULL, - NULL, - NULL -}; - - -PyMODINIT_FUNC -PyInit__multiprocessing(void) +static int +multiprocessing_exec(PyObject *module) { - PyObject *module, *temp, *value = NULL; - - /* Initialize module */ - module = PyModule_Create(&multiprocessing_module); - if (!module) - return NULL; - #if defined(MS_WINDOWS) || \ (defined(HAVE_SEM_OPEN) && !defined(POSIX_SEMAPHORES_NOT_ENABLED)) + /* Add _PyMp_SemLock type to module */ - if (PyType_Ready(&_PyMp_SemLockType) < 0) - return NULL; - Py_INCREF(&_PyMp_SemLockType); + if (PyModule_AddType(module, &_PyMp_SemLockType) < 0) { + return -1; + } + { PyObject *py_sem_value_max; /* Some systems define SEM_VALUE_MAX as an unsigned value that @@ -223,25 +205,41 @@ PyInit__multiprocessing(void) py_sem_value_max = PyLong_FromLong(INT_MAX); else py_sem_value_max = PyLong_FromLong(SEM_VALUE_MAX); - if (py_sem_value_max == NULL) - return NULL; - PyDict_SetItemString(_PyMp_SemLockType.tp_dict, "SEM_VALUE_MAX", - py_sem_value_max); + + if (py_sem_value_max == NULL) { + Py_DECREF(py_sem_value_max); + return -1; + } + if (PyDict_SetItemString(_PyMp_SemLockType.tp_dict, "SEM_VALUE_MAX", + py_sem_value_max) < 0) { + Py_DECREF(py_sem_value_max); + return -1; + } + Py_DECREF(py_sem_value_max); } - PyModule_AddObject(module, "SemLock", (PyObject*)&_PyMp_SemLockType); + #endif /* Add configuration macros */ - temp = PyDict_New(); - if (!temp) - return NULL; + PyObject *flags = PyDict_New(); + if (!flags) { + return -1; + } -#define ADD_FLAG(name) \ - value = Py_BuildValue("i", name); \ - if (value == NULL) { Py_DECREF(temp); return NULL; } \ - if (PyDict_SetItemString(temp, #name, value) < 0) { \ - Py_DECREF(temp); Py_DECREF(value); return NULL; } \ - Py_DECREF(value) +#define ADD_FLAG(name) \ + do { \ + PyObject *value = PyLong_FromLong(name); \ + if (value == NULL) { \ + Py_DECREF(flags); \ + return -1; \ + } \ + if (PyDict_SetItemString(flags, #name, value) < 0) { \ + Py_DECREF(flags); \ + Py_DECREF(value); \ + return -1; \ + } \ + Py_DECREF(value); \ + } while (0) #if defined(HAVE_SEM_OPEN) && !defined(POSIX_SEMAPHORES_NOT_ENABLED) ADD_FLAG(HAVE_SEM_OPEN); @@ -256,8 +254,28 @@ PyInit__multiprocessing(void) ADD_FLAG(HAVE_BROKEN_SEM_UNLINK); #endif - if (PyModule_AddObject(module, "flags", temp) < 0) - return NULL; + if (PyModule_AddObject(module, "flags", flags) < 0) { + Py_DECREF(flags); + return -1; + } + + return 0; +} + +static PyModuleDef_Slot multiprocessing_slots[] = { + {Py_mod_exec, multiprocessing_exec}, + {0, NULL} +}; - return module; +static struct PyModuleDef multiprocessing_module = { + PyModuleDef_HEAD_INIT, + .m_name = "_multiprocessing", + .m_methods = module_methods, + .m_slots = multiprocessing_slots, +}; + +PyMODINIT_FUNC +PyInit__multiprocessing(void) +{ + return PyModuleDef_Init(&multiprocessing_module); } From webhook-mailer at python.org Tue Aug 11 09:27:08 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 11 Aug 2020 13:27:08 -0000 Subject: [Python-checkins] bpo-41521: Replace whitelist/blacklist with allowlist/denylist (GH-21822) Message-ID: https://github.com/python/cpython/commit/fabd7bb8e0450f16ed5c5c0ad575aa413d65712d commit: fabd7bb8e0450f16ed5c5c0ad575aa413d65712d branch: master author: Victor Stinner committer: GitHub date: 2020-08-11T06:26:59-07:00 summary: bpo-41521: Replace whitelist/blacklist with allowlist/denylist (GH-21822) Automerge-Triggered-By: @tiran files: M Doc/library/http.cookiejar.rst M Doc/library/urllib.parse.rst M Lib/codecs.py M Lib/ipaddress.py M Lib/test/test___all__.py M Lib/test/test_httplib.py M Lib/test/test_httpservers.py M Lib/test/test_nntplib.py M Lib/test/test_tools/test_sundry.py M Lib/test/test_traceback.py M Objects/typeobject.c M Tools/clinic/clinic.py diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 9ac5d52a2ab09..0a9ee7eb516f4 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -462,16 +462,16 @@ receiving cookies. There are also some strictness switches that allow you to tighten up the rather loose Netscape protocol rules a little bit (at the cost of blocking some benign cookies). -A domain blacklist and whitelist is provided (both off by default). Only domains -not in the blacklist and present in the whitelist (if the whitelist is active) +A domain denylist and allowlist is provided (both off by default). Only domains +not in the denylist and present in the allowlist (if the allowlist is active) participate in cookie setting and returning. Use the *blocked_domains* constructor argument, and :meth:`blocked_domains` and :meth:`set_blocked_domains` methods (and the corresponding argument and methods -for *allowed_domains*). If you set a whitelist, you can turn it off again by +for *allowed_domains*). If you set an allowlist, you can turn it off again by setting it to :const:`None`. Domains in block or allow lists that do not start with a dot must equal the -cookie domain to be matched. For example, ``"example.com"`` matches a blacklist +cookie domain to be matched. For example, ``"example.com"`` matches a denylist entry of ``"example.com"``, but ``"www.example.com"`` does not. Domains that do start with a dot are matched by more specific domains too. For example, both ``"www.example.com"`` and ``"www.coyote.example.com"`` match ``".example.com"`` @@ -494,7 +494,7 @@ and ``".168.1.2"``, 192.168.1.2 is blocked, but 193.168.1.2 is not. .. method:: DefaultCookiePolicy.is_blocked(domain) - Return whether *domain* is on the blacklist for setting or receiving cookies. + Return whether *domain* is on the denylist for setting or receiving cookies. .. method:: DefaultCookiePolicy.allowed_domains() @@ -509,7 +509,7 @@ and ``".168.1.2"``, 192.168.1.2 is blocked, but 193.168.1.2 is not. .. method:: DefaultCookiePolicy.is_not_allowed(domain) - Return whether *domain* is not on the whitelist for setting or receiving + Return whether *domain* is not on the allowlist for setting or receiving cookies. :class:`DefaultCookiePolicy` instances have the following attributes, which are diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index 536cf952bda43..f9c8ba7398f66 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -153,7 +153,7 @@ or on combining URL components into a URL string. .. versionchanged:: 3.3 The fragment is now parsed for all URL schemes (unless *allow_fragment* is - false), in accordance with :rfc:`3986`. Previously, a whitelist of + false), in accordance with :rfc:`3986`. Previously, an allowlist of schemes that support fragments existed. .. versionchanged:: 3.6 diff --git a/Lib/codecs.py b/Lib/codecs.py index 7f23e9775df80..3935490d88c8d 100644 --- a/Lib/codecs.py +++ b/Lib/codecs.py @@ -83,7 +83,7 @@ class CodecInfo(tuple): """Codec details when looking up the codec registry""" - # Private API to allow Python 3.4 to blacklist the known non-Unicode + # Private API to allow Python 3.4 to denylist the known non-Unicode # codecs in the standard library. A more general mechanism to # reliably distinguish test encodings from other codecs will hopefully # be defined for Python 3.5 diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index bc662c415b2a4..160b16dbc162f 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1214,7 +1214,7 @@ def _parse_octet(cls, octet_str): """ if not octet_str: raise ValueError("Empty octet not permitted") - # Whitelist the characters, since int() allows a lot of bizarre stuff. + # Reject non-ASCII digits. if not (octet_str.isascii() and octet_str.isdigit()): msg = "Only decimal digits permitted in %r" raise ValueError(msg % octet_str) @@ -1719,7 +1719,7 @@ def _parse_hextet(cls, hextet_str): [0..FFFF]. """ - # Whitelist the characters, since int() allows a lot of bizarre stuff. + # Reject non-ASCII digits. if not cls._HEX_DIGITS.issuperset(hextet_str): raise ValueError("Only hex digits permitted in %r" % hextet_str) # We do the length check second, since the invalid character error diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py index 0a03dd20065fe..c6ce64864c0db 100644 --- a/Lib/test/test___all__.py +++ b/Lib/test/test___all__.py @@ -69,8 +69,8 @@ def walk_modules(self, basedir, modpath): yield path, modpath + fn[:-3] def test_all(self): - # Blacklisted modules and packages - blacklist = set([ + # List of denied modules and packages + denylist = set([ # Will raise a SyntaxError when compiling the exec statement '__future__', ]) @@ -85,13 +85,13 @@ def test_all(self): lib_dir = os.path.dirname(os.path.dirname(__file__)) for path, modname in self.walk_modules(lib_dir, ""): m = modname - blacklisted = False + denylisted = False while m: - if m in blacklist: - blacklisted = True + if m in denylist: + denylisted = True break m = m.rpartition('.')[0] - if blacklisted: + if denylisted: continue if support.verbose: print(modname) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py index 3431bb80ea6de..a3f268be97921 100644 --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -1434,9 +1434,9 @@ def test_all(self): expected = {"responses"} # White-list documented dict() object # HTTPMessage, parse_headers(), and the HTTP status code constants are # intentionally omitted for simplicity - blacklist = {"HTTPMessage", "parse_headers"} + denylist = {"HTTPMessage", "parse_headers"} for name in dir(client): - if name.startswith("_") or name in blacklist: + if name.startswith("_") or name in denylist: continue module_object = getattr(client, name) if getattr(module_object, "__module__", None) == "http.client": diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py index a7e1719ab60ee..2859abb21fc9f 100644 --- a/Lib/test/test_httpservers.py +++ b/Lib/test/test_httpservers.py @@ -1189,9 +1189,9 @@ def test_windows_colon(self): class MiscTestCase(unittest.TestCase): def test_all(self): expected = [] - blacklist = {'executable', 'nobody_uid', 'test'} + denylist = {'executable', 'nobody_uid', 'test'} for name in dir(server): - if name.startswith('_') or name in blacklist: + if name.startswith('_') or name in denylist: continue module_object = getattr(server, name) if getattr(module_object, '__module__', None) == 'http.server': diff --git a/Lib/test/test_nntplib.py b/Lib/test/test_nntplib.py index 1df64fa7c6b00..b11c19c84d3fb 100644 --- a/Lib/test/test_nntplib.py +++ b/Lib/test/test_nntplib.py @@ -197,11 +197,11 @@ def test_article_head_body(self): self.assertTrue(resp.startswith("220 "), resp) self.check_article_resp(resp, article, art_num) # Tolerate running the tests from behind a NNTP virus checker - blacklist = lambda line: line.startswith(b'X-Antivirus') + denylist = lambda line: line.startswith(b'X-Antivirus') filtered_head_lines = [line for line in head.lines - if not blacklist(line)] + if not denylist(line)] filtered_lines = [line for line in article.lines - if not blacklist(line)] + if not denylist(line)] self.assertEqual(filtered_lines, filtered_head_lines + [b''] + body.lines) def test_capabilities(self): diff --git a/Lib/test/test_tools/test_sundry.py b/Lib/test/test_tools/test_sundry.py index 8b5a963e25bd1..52369ec09a77f 100644 --- a/Lib/test/test_tools/test_sundry.py +++ b/Lib/test/test_tools/test_sundry.py @@ -16,18 +16,18 @@ class TestSundryScripts(unittest.TestCase): # At least make sure the rest don't have syntax errors. When tests are - # added for a script it should be added to the whitelist below. + # added for a script it should be added to the allowlist below. # scripts that have independent tests. - whitelist = ['reindent', 'pdeps', 'gprof2html', 'md5sum'] + allowlist = ['reindent', 'pdeps', 'gprof2html', 'md5sum'] # scripts that can't be imported without running - blacklist = ['make_ctype'] + denylist = ['make_ctype'] # scripts that use windows-only modules windows_only = ['win_add2path'] - # blacklisted for other reasons + # denylisted for other reasons other = ['analyze_dxp', '2to3'] - skiplist = blacklist + whitelist + windows_only + other + skiplist = denylist + allowlist + windows_only + other def test_sundry(self): old_modules = import_helper.modules_setup() diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index c5fbd8700ae9b..730596efd8bce 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -1191,9 +1191,9 @@ class MiscTest(unittest.TestCase): def test_all(self): expected = set() - blacklist = {'print_list'} + denylist = {'print_list'} for name in dir(traceback): - if name.startswith('_') or name in blacklist: + if name.startswith('_') or name in denylist: continue module_object = getattr(traceback, name) if getattr(module_object, '__module__', None) == 'traceback': diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 49b3c859e35c1..c66f8fcec8ed5 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -4171,10 +4171,10 @@ object_set_class(PyObject *self, PyObject *value, void *closure) In theory the proper fix would be to identify which classes rely on this invariant and somehow disallow __class__ assignment only for them, perhaps via some mechanism like a new Py_TPFLAGS_IMMUTABLE flag (a - "blacklisting" approach). But in practice, since this problem wasn't + "denylisting" approach). But in practice, since this problem wasn't noticed late in the 3.5 RC cycle, we're taking the conservative approach and reinstating the same HEAPTYPE->HEAPTYPE check that we used - to have, plus a "whitelist". For now, the whitelist consists only of + to have, plus an "allowlist". For now, the allowlist consists only of ModuleType subtypes, since those are the cases that motivated the patch in the first place -- see https://bugs.python.org/issue22986 -- and since module objects are mutable we can be sure that they are diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py index 92334d9195fcc..1bbbd4f9fb193 100755 --- a/Tools/clinic/clinic.py +++ b/Tools/clinic/clinic.py @@ -4433,7 +4433,7 @@ def state_parameter(self, line): if 'c_default' not in kwargs: # we can only represent very simple data values in C. - # detect whether default is okay, via a blacklist + # detect whether default is okay, via a denylist # of disallowed ast nodes. class DetectBadNodes(ast.NodeVisitor): bad = False @@ -4456,9 +4456,9 @@ def bad_node(self, node): # "starred": "a = [1, 2, 3]; *a" visit_Starred = bad_node - blacklist = DetectBadNodes() - blacklist.visit(module) - bad = blacklist.bad + denylist = DetectBadNodes() + denylist.visit(module) + bad = denylist.bad else: # if they specify a c_default, we can be more lenient about the default value. # but at least make an attempt at ensuring it's a valid expression. From webhook-mailer at python.org Tue Aug 11 09:28:51 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Tue, 11 Aug 2020 13:28:51 -0000 Subject: [Python-checkins] bpo-41521: Replace whitelist/blacklist with allowlist/denylist (GH-21823) Message-ID: https://github.com/python/cpython/commit/0ee0b2938cb606151d8d287472c838044bad4a0e commit: 0ee0b2938cb606151d8d287472c838044bad4a0e branch: master author: Victor Stinner committer: GitHub date: 2020-08-11T15:28:43+02:00 summary: bpo-41521: Replace whitelist/blacklist with allowlist/denylist (GH-21823) Rename 5 test method names in test_codecs and test_typing. files: M Lib/test/test_codecs.py M Lib/test/test_typing.py diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index f0da35c039e11..3dd56820cd107 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2644,7 +2644,7 @@ def test_buffer_api_usage(self): view_decoded = codecs.decode(view, encoding) self.assertEqual(view_decoded, data) - def test_text_to_binary_blacklists_binary_transforms(self): + def test_text_to_binary_denylists_binary_transforms(self): # Check binary -> binary codecs give a good error for str input bad_input = "bad input type" for encoding in bytes_transform_encodings: @@ -2656,14 +2656,14 @@ def test_text_to_binary_blacklists_binary_transforms(self): bad_input.encode(encoding) self.assertIsNone(failure.exception.__cause__) - def test_text_to_binary_blacklists_text_transforms(self): + def test_text_to_binary_denylists_text_transforms(self): # Check str.encode gives a good error message for str -> str codecs msg = (r"^'rot_13' is not a text encoding; " r"use codecs.encode\(\) to handle arbitrary codecs") with self.assertRaisesRegex(LookupError, msg): "just an example message".encode("rot_13") - def test_binary_to_text_blacklists_binary_transforms(self): + def test_binary_to_text_denylists_binary_transforms(self): # Check bytes.decode and bytearray.decode give a good error # message for binary -> binary codecs data = b"encode first to ensure we meet any format restrictions" @@ -2678,7 +2678,7 @@ def test_binary_to_text_blacklists_binary_transforms(self): with self.assertRaisesRegex(LookupError, msg): bytearray(encoded_data).decode(encoding) - def test_binary_to_text_blacklists_text_transforms(self): + def test_binary_to_text_denylists_text_transforms(self): # Check str -> str codec gives a good error for binary input for bad_input in (b"immutable", bytearray(b"mutable")): with self.subTest(bad_input=bad_input): diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 7f96aff710455..b3be99141afca 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1387,7 +1387,7 @@ def close(self): self.assertIsSubclass(B, Custom) self.assertNotIsSubclass(A, Custom) - def test_builtin_protocol_whitelist(self): + def test_builtin_protocol_allowlist(self): with self.assertRaises(TypeError): class CustomProtocol(TestCase, Protocol): pass From webhook-mailer at python.org Tue Aug 11 10:08:10 2020 From: webhook-mailer at python.org (Edward K. Ream) Date: Tue, 11 Aug 2020 14:08:10 -0000 Subject: [Python-checkins] Add links to asttokens, leoAst, LibCST and parso to ast docs (GH-21773) Message-ID: https://github.com/python/cpython/commit/e3c971ccfa58afcb2656b71b95e10b9703f2ad32 commit: e3c971ccfa58afcb2656b71b95e10b9703f2ad32 branch: master author: Edward K. Ream committer: GitHub date: 2020-08-11T07:07:49-07:00 summary: Add links to asttokens, leoAst, LibCST and parso to ast docs (GH-21773) files: M Doc/library/ast.rst diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 25cb17811e718..755c60fba6411 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -1800,5 +1800,24 @@ to stdout. Otherwise, the content is read from stdin. .. seealso:: - `Green Tree Snakes `_, an external documentation resource, has good - details on working with Python ASTs. + `Green Tree Snakes `_, an external + documentation resource, has good details on working with Python ASTs. + + `ASTTokens `_ + annotates Python ASTs with the positions of tokens and text in the source + code that generated them. This is helpful for tools that make source code + transformations. + + `leoAst.py `_ unifies the + token-based and parse-tree-based views of python programs by inserting + two-way links between tokens and ast nodes. + + `LibCST `_ parses code as a Concrete Syntax + Tree that looks like an ast tree and keeps all formatting details. It's + useful for building automated refactoring (codemod) applications and + linters. + + `Parso `_ is a Python parser that supports + error recovery and round-trip parsing for different Python versions (in + multiple Python versions). Parso is also able to list multiple syntax errors + in your python file. \ No newline at end of file From webhook-mailer at python.org Tue Aug 11 10:45:31 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Tue, 11 Aug 2020 14:45:31 -0000 Subject: [Python-checkins] [3.8] bpo-41504: Add links to asttokens, leoAst, LibCST and parso to ast docs (GH-21773) (GH-21829) Message-ID: https://github.com/python/cpython/commit/7b3ceaa71051d811bfee0b62ef9cb0fd2121c7ee commit: 7b3ceaa71051d811bfee0b62ef9cb0fd2121c7ee branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-11T07:45:25-07:00 summary: [3.8] bpo-41504: Add links to asttokens, leoAst, LibCST and parso to ast docs (GH-21773) (GH-21829) (cherry picked from commit e3c971ccfa58afcb2656b71b95e10b9703f2ad32) Co-authored-by: Edward K. Ream https://bugs.python.org/issue41504 files: M Doc/library/ast.rst diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 6ca27609e93ba..67c6392ad7d2c 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -343,5 +343,24 @@ and classes for traversing abstract syntax trees: .. seealso:: - `Green Tree Snakes `_, an external documentation resource, has good - details on working with Python ASTs. + `Green Tree Snakes `_, an external + documentation resource, has good details on working with Python ASTs. + + `ASTTokens `_ + annotates Python ASTs with the positions of tokens and text in the source + code that generated them. This is helpful for tools that make source code + transformations. + + `leoAst.py `_ unifies the + token-based and parse-tree-based views of python programs by inserting + two-way links between tokens and ast nodes. + + `LibCST `_ parses code as a Concrete Syntax + Tree that looks like an ast tree and keeps all formatting details. It's + useful for building automated refactoring (codemod) applications and + linters. + + `Parso `_ is a Python parser that supports + error recovery and round-trip parsing for different Python versions (in + multiple Python versions). Parso is also able to list multiple syntax errors + in your python file. \ No newline at end of file From webhook-mailer at python.org Tue Aug 11 12:16:26 2020 From: webhook-mailer at python.org (Petr Viktorin) Date: Tue, 11 Aug 2020 16:16:26 -0000 Subject: [Python-checkins] Add PEP 573 additions to What's New (GH-21374) Message-ID: https://github.com/python/cpython/commit/af3a6a8caefc2b202c831ab908677c1a4371cc27 commit: af3a6a8caefc2b202c831ab908677c1a4371cc27 branch: master author: Petr Viktorin committer: GitHub date: 2020-08-11T18:15:57+02:00 summary: Add PEP 573 additions to What's New (GH-21374) files: M Doc/whatsnew/3.9.rst diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst index db380bc1cdfa4..be7406e13c2cd 100644 --- a/Doc/whatsnew/3.9.rst +++ b/Doc/whatsnew/3.9.rst @@ -1024,6 +1024,13 @@ C API Changes New Features ------------ +* :pep:`573`: Add :c:func:`PyType_FromModuleAndSpec` to associate + a module with a class; :c:func:`PyType_GetModule` and + :c:func:`PyType_GetModuleState` to retrieve the module and its state; and + :c:data:`PyCMethod` and :c:data:`METH_METHOD` to allow a method to + access the class it was defined in. + (Contributed by Marcel Plch and Petr Viktorin in :issue:`38787`.) + * Add :c:func:`PyFrame_GetCode` function: get a frame code. Add :c:func:`PyFrame_GetBack` function: get the frame next outer frame. (Contributed by Victor Stinner in :issue:`40421`.) From webhook-mailer at python.org Tue Aug 11 12:33:30 2020 From: webhook-mailer at python.org (Ram Rachum) Date: Tue, 11 Aug 2020 16:33:30 -0000 Subject: [Python-checkins] bpo-41475: Fix note in "What's new in 3.7" (#21733) Message-ID: https://github.com/python/cpython/commit/76643c10ede2813ca921464fe839e81caee21a84 commit: 76643c10ede2813ca921464fe839e81caee21a84 branch: master author: Ram Rachum committer: GitHub date: 2020-08-11T18:33:25+02:00 summary: bpo-41475: Fix note in "What's new in 3.7" (#21733) files: M Doc/whatsnew/3.7.rst diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 59b96621bdd4b..279bbc697b5c6 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -171,7 +171,7 @@ on a per-module basis in Python 3.7 using a :mod:`__future__` import:: from __future__ import annotations -It will become the default in Python 4.0. +It will become the default in Python 3.10. .. seealso:: From webhook-mailer at python.org Tue Aug 11 15:15:00 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Tue, 11 Aug 2020 19:15:00 -0000 Subject: [Python-checkins] Call randseed() before other imports in deccheck.py (GH-21834) Message-ID: https://github.com/python/cpython/commit/b5f87b93a542082551c67538523d318f0d46e16e commit: b5f87b93a542082551c67538523d318f0d46e16e branch: master author: Stefan Krah committer: GitHub date: 2020-08-11T21:14:51+02:00 summary: Call randseed() before other imports in deccheck.py (GH-21834) files: M Modules/_decimal/tests/deccheck.py diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index 0b2a1c49336ef..ca869f4dbf5d8 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -30,10 +30,14 @@ # +import random +import time + +RANDSEED = int(time.time()) +random.seed(RANDSEED) + import sys import os -import time -import random from copy import copy from collections import defaultdict @@ -1235,10 +1239,6 @@ def check_untested(funcdict, c_cls, p_cls): args.single = args.single[0] - randseed = int(time.time()) - random.seed(randseed) - - # Set up the testspecs list. A testspec is simply a dictionary # that determines the amount of different contexts that 'test_method' # will generate. @@ -1306,9 +1306,9 @@ def check_untested(funcdict, c_cls, p_cls): if args.multicore: q = Queue() elif args.single: - log("Random seed: %d", randseed) + log("Random seed: %d", RANDSEED) else: - log("\n\nRandom seed: %d\n\n", randseed) + log("\n\nRandom seed: %d\n\n", RANDSEED) FOUND_METHOD = False From webhook-mailer at python.org Tue Aug 11 18:27:16 2020 From: webhook-mailer at python.org (Christopher Yeh) Date: Tue, 11 Aug 2020 22:27:16 -0000 Subject: [Python-checkins] Fix typo (GH-21820) Message-ID: https://github.com/python/cpython/commit/0dfee33dfe6b4683016dc920ee934d3a2d7323a1 commit: 0dfee33dfe6b4683016dc920ee934d3a2d7323a1 branch: master author: Christopher Yeh committer: GitHub date: 2020-08-11T19:27:08-03:00 summary: Fix typo (GH-21820) files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 7028d240c59eb..5a10faa7bbd29 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4205,7 +4205,7 @@ The constructors for both classes work the same: Note, the non-operator versions of :meth:`union`, :meth:`intersection`, - :meth:`difference`, and :meth:`symmetric_difference`, :meth:`issubset`, and + :meth:`difference`, :meth:`symmetric_difference`, :meth:`issubset`, and :meth:`issuperset` methods will accept any iterable as an argument. In contrast, their operator based counterparts require their arguments to be sets. This precludes error-prone constructions like ``set('abc') & 'cbs'`` From webhook-mailer at python.org Wed Aug 12 04:53:25 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 12 Aug 2020 08:53:25 -0000 Subject: [Python-checkins] bpo-41521, typing: Rename _PROTO_WHITELIST to _PROTO_ALLOWLIST (#21825) Message-ID: https://github.com/python/cpython/commit/0e95bbf08571e98f4b688524efc2dcf20d315d91 commit: 0e95bbf08571e98f4b688524efc2dcf20d315d91 branch: master author: Victor Stinner committer: GitHub date: 2020-08-12T10:53:12+02:00 summary: bpo-41521, typing: Rename _PROTO_WHITELIST to _PROTO_ALLOWLIST (#21825) files: M Lib/typing.py diff --git a/Lib/typing.py b/Lib/typing.py index 5da032bbee8f1..fce8da4fe3cf0 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1021,7 +1021,7 @@ def _allow_reckless_class_cheks(): return True -_PROTO_WHITELIST = { +_PROTO_ALLOWLIST = { 'collections.abc': [ 'Callable', 'Awaitable', 'Iterable', 'Iterator', 'AsyncIterable', 'Hashable', 'Sized', 'Container', 'Collection', 'Reversible', @@ -1140,8 +1140,8 @@ def _proto_hook(other): # ... otherwise check consistency of bases, and prohibit instantiation. for base in cls.__bases__: if not (base in (object, Generic) or - base.__module__ in _PROTO_WHITELIST and - base.__name__ in _PROTO_WHITELIST[base.__module__] or + base.__module__ in _PROTO_ALLOWLIST and + base.__name__ in _PROTO_ALLOWLIST[base.__module__] or issubclass(base, Generic) and base._is_protocol): raise TypeError('Protocols can only inherit from other' ' protocols, got %r' % base) From webhook-mailer at python.org Wed Aug 12 06:36:25 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 12 Aug 2020 10:36:25 -0000 Subject: [Python-checkins] bpo-41475: Fix note in "What's new in 3.7" (GH-21733) (GH-21835) Message-ID: https://github.com/python/cpython/commit/a8ad127c222456e614b59990f113e93e95593155 commit: a8ad127c222456e614b59990f113e93e95593155 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-12T06:36:20-04:00 summary: bpo-41475: Fix note in "What's new in 3.7" (GH-21733) (GH-21835) (cherry picked from commit 76643c10ede2813ca921464fe839e81caee21a84) Co-authored-by: Ram Rachum files: M Doc/whatsnew/3.7.rst diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 9644a4f31a957..6dcb006924e77 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -171,7 +171,7 @@ on a per-module basis in Python 3.7 using a :mod:`__future__` import:: from __future__ import annotations -It will become the default in Python 4.0. +It will become the default in Python 3.10. .. seealso:: From webhook-mailer at python.org Wed Aug 12 06:38:41 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 12 Aug 2020 10:38:41 -0000 Subject: [Python-checkins] bpo-41475: Fix note in "What's new in 3.7" (GH-21733) (GH-21833) Message-ID: https://github.com/python/cpython/commit/622d90f65ca9f0a6ddf255a727de003b92dca01d commit: 622d90f65ca9f0a6ddf255a727de003b92dca01d branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-12T06:38:37-04:00 summary: bpo-41475: Fix note in "What's new in 3.7" (GH-21733) (GH-21833) (cherry picked from commit 76643c10ede2813ca921464fe839e81caee21a84) Co-authored-by: Ram Rachum files: M Doc/whatsnew/3.7.rst diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index b9b50216d23a6..013f14dd9da16 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -171,7 +171,7 @@ on a per-module basis in Python 3.7 using a :mod:`__future__` import:: from __future__ import annotations -It will become the default in Python 4.0. +It will become the default in Python 3.10. .. seealso:: From webhook-mailer at python.org Wed Aug 12 06:53:29 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 12 Aug 2020 10:53:29 -0000 Subject: [Python-checkins] bpo-41455: Provide a link to how the third generation is collected in the GC docs (GH-21703) (GH-21788) Message-ID: https://github.com/python/cpython/commit/f3b6f3cd9ac6931ae346cf298fae7b691d5656bb commit: f3b6f3cd9ac6931ae346cf298fae7b691d5656bb branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-12T06:53:13-04:00 summary: bpo-41455: Provide a link to how the third generation is collected in the GC docs (GH-21703) (GH-21788) Co-authored-by: Pablo Galindo (cherry picked from commit 82ca8fada15b121866530f2cdac1b7055be4a244) Co-authored-by: Yaroslav Pankovych <31005942+P-Alban at users.noreply.github.com> files: M Doc/library/gc.rst diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index af45581e33527..1f09ed52ce867 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -103,9 +103,9 @@ The :mod:`gc` module provides the following functions: allocations minus the number of deallocations exceeds *threshold0*, collection starts. Initially only generation ``0`` is examined. If generation ``0`` has been examined more than *threshold1* times since generation ``1`` has been - examined, then generation ``1`` is examined as well. Similarly, *threshold2* - controls the number of collections of generation ``1`` before collecting - generation ``2``. + examined, then generation ``1`` is examined as well. + With the third generation, things are a bit more complicated, + see `Collecting the oldest generation `_ for more information. .. function:: get_count() From webhook-mailer at python.org Wed Aug 12 08:53:59 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 12 Aug 2020 12:53:59 -0000 Subject: [Python-checkins] bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838) Message-ID: https://github.com/python/cpython/commit/369a1cbdee14d9f27356fb3a8bb21e4fde289d25 commit: 369a1cbdee14d9f27356fb3a8bb21e4fde289d25 branch: master author: Victor Stinner committer: GitHub date: 2020-08-12T14:53:28+02:00 summary: bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838) files: A Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst M Lib/codeop.py M Lib/test/test_codeop.py diff --git a/Lib/codeop.py b/Lib/codeop.py index 7e192ea6a10a0..547629262d068 100644 --- a/Lib/codeop.py +++ b/Lib/codeop.py @@ -84,9 +84,11 @@ def _maybe_compile(compiler, source, filename, symbol): except SyntaxError: pass - # Suppress warnings after the first compile to avoid duplication. + # Catch syntax warnings after the first compile + # to emit SyntaxWarning at most once. with warnings.catch_warnings(): - warnings.simplefilter("ignore") + warnings.simplefilter("error", SyntaxWarning) + try: code1 = compiler(source + "\n", filename, symbol) except SyntaxError as e: diff --git a/Lib/test/test_codeop.py b/Lib/test/test_codeop.py index 6e821df6b0e70..7984e5f1e5a69 100644 --- a/Lib/test/test_codeop.py +++ b/Lib/test/test_codeop.py @@ -4,6 +4,7 @@ """ import sys import unittest +import warnings from test import support from test.support import warnings_helper @@ -310,5 +311,11 @@ def test_warning(self): compile_command("0 is 0") self.assertEqual(len(w.warnings), 1) + # bpo-41520: check SyntaxWarning treated as an SyntaxError + with self.assertRaises(SyntaxError): + warnings.simplefilter('error', SyntaxWarning) + compile_command('1 is 1\n', symbol='exec') + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst new file mode 100644 index 0000000000000..ca5501c2aeec0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst @@ -0,0 +1 @@ +Fix :mod:`codeop` regression: it no longer ignores :exc:`SyntaxWarning`. From webhook-mailer at python.org Wed Aug 12 09:12:10 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 12 Aug 2020 13:12:10 -0000 Subject: [Python-checkins] bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838) Message-ID: https://github.com/python/cpython/commit/afff51fc09993dff1693aacb440221314b163409 commit: afff51fc09993dff1693aacb440221314b163409 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-12T06:12:05-07:00 summary: bpo-41520: codeop no longer ignores SyntaxWarning (GH-21838) (cherry picked from commit 369a1cbdee14d9f27356fb3a8bb21e4fde289d25) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst M Lib/codeop.py M Lib/test/test_codeop.py diff --git a/Lib/codeop.py b/Lib/codeop.py index 3c2bb6083561e..97043877d1869 100644 --- a/Lib/codeop.py +++ b/Lib/codeop.py @@ -84,9 +84,11 @@ def _maybe_compile(compiler, source, filename, symbol): except SyntaxError as err: pass - # Suppress warnings after the first compile to avoid duplication. + # Catch syntax warnings after the first compile + # to emit SyntaxWarning at most once. with warnings.catch_warnings(): - warnings.simplefilter("ignore") + warnings.simplefilter("error", SyntaxWarning) + try: code1 = compiler(source + "\n", filename, symbol) except SyntaxError as e: diff --git a/Lib/test/test_codeop.py b/Lib/test/test_codeop.py index 1e57ab9d51e2c..1eae26b6ce17e 100644 --- a/Lib/test/test_codeop.py +++ b/Lib/test/test_codeop.py @@ -3,6 +3,7 @@ Nick Mathewson """ import unittest +import warnings from test import support from codeop import compile_command, PyCF_DONT_IMPLY_DEDENT @@ -300,5 +301,11 @@ def test_warning(self): compile_command("0 is 0") self.assertEqual(len(w.warnings), 1) + # bpo-41520: check SyntaxWarning treated as an SyntaxError + with self.assertRaises(SyntaxError): + warnings.simplefilter('error', SyntaxWarning) + compile_command('1 is 1\n', symbol='exec') + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst new file mode 100644 index 0000000000000..ca5501c2aeec0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst @@ -0,0 +1 @@ +Fix :mod:`codeop` regression: it no longer ignores :exc:`SyntaxWarning`. From webhook-mailer at python.org Wed Aug 12 10:00:16 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Wed, 12 Aug 2020 14:00:16 -0000 Subject: [Python-checkins] Catch all skip_handler cases (GH-21842) Message-ID: https://github.com/python/cpython/commit/6e0b7888815e74c67b1fc3c5f60dd4a1aeae127a commit: 6e0b7888815e74c67b1fc3c5f60dd4a1aeae127a branch: master author: Stefan Krah committer: GitHub date: 2020-08-12T16:00:05+02:00 summary: Catch all skip_handler cases (GH-21842) files: M Modules/_decimal/tests/deccheck.py diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index ca869f4dbf5d8..5de57d1f875a2 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -185,7 +185,7 @@ def p_as_triple(dec): coeff = int(s) if s else 0 if coeff < 0 or coeff >= 2**128: - raise ValueError("value out of bounds for a uint128 triple"); + raise ValueError("value out of bounds for a uint128 triple") return (sign, coeff, exp) @@ -193,7 +193,7 @@ def p_from_triple(triple): sign, coeff, exp = triple if coeff < 0 or coeff >= 2**128: - raise ValueError("value out of bounds for a uint128 triple"); + raise ValueError("value out of bounds for a uint128 triple") digits = tuple(int(c) for c in str(coeff)) @@ -894,7 +894,7 @@ def verify(t, stat): t.presults.append(str(t.rp.real)) ctriple = None - if t.funcname not in ['__radd__', '__rmul__']: # see skip handler + if str(t.rc) == str(t.rp): # see skip handler try: ctriple = c_as_triple(t.rc) except ValueError: From webhook-mailer at python.org Wed Aug 12 15:49:50 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Wed, 12 Aug 2020 19:49:50 -0000 Subject: [Python-checkins] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) Message-ID: https://github.com/python/cpython/commit/423e77d6de497931585d1883805a9e3fa4096b0b commit: 423e77d6de497931585d1883805a9e3fa4096b0b branch: master author: Victor Stinner committer: GitHub date: 2020-08-12T21:49:22+02:00 summary: bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) Enable Sphinx 3.2 "c_allow_pre_v3" option and disable the c_warn_on_allowed_pre_v3 option to make the documentation compatible with Sphinx 2 and Sphinx 3. files: A Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst M Doc/conf.py diff --git a/Doc/conf.py b/Doc/conf.py index bfb2a98fc63a2..079d17717f381 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -228,3 +228,13 @@ # Relative filename of the reference count data file. refcount_file = 'data/refcounts.dat' + +# Sphinx 2 and Sphinx 3 compatibility +# ----------------------------------- + +# bpo-40204: Allow Sphinx 2 syntax in the C domain +c_allow_pre_v3 = True + +# bpo-40204: Disable warnings on Sphinx 2 syntax of the C domain since the +# documentation is built with -W (warnings treated as errors). +c_warn_on_allowed_pre_v3 = False diff --git a/Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst b/Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst new file mode 100644 index 0000000000000..152f6c98b9004 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst @@ -0,0 +1,3 @@ +Enable Sphinx 3.2 ``c_allow_pre_v3`` option and disable +``c_warn_on_allowed_pre_v3`` option to make the documentation compatible +with Sphinx 2 and Sphinx 3. From webhook-mailer at python.org Wed Aug 12 17:23:38 2020 From: webhook-mailer at python.org (Hai Shi) Date: Wed, 12 Aug 2020 21:23:38 -0000 Subject: [Python-checkins] bpo-1635741: Clean sysdict and builtins of interpreter at exit (GH-21605) Message-ID: https://github.com/python/cpython/commit/8ecc0c4d390d03de5cd2344aa44b69ed02ffe470 commit: 8ecc0c4d390d03de5cd2344aa44b69ed02ffe470 branch: master author: Hai Shi committer: GitHub date: 2020-08-12T23:23:30+02:00 summary: bpo-1635741: Clean sysdict and builtins of interpreter at exit (GH-21605) files: M Python/pystate.c diff --git a/Python/pystate.c b/Python/pystate.c index d0cbf5cb8364b..f6d1956e9dce9 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -294,8 +294,6 @@ PyInterpreterState_Clear(PyInterpreterState *interp) Py_CLEAR(interp->codec_error_registry); Py_CLEAR(interp->modules); Py_CLEAR(interp->modules_by_index); - Py_CLEAR(interp->sysdict); - Py_CLEAR(interp->builtins); Py_CLEAR(interp->builtins_copy); Py_CLEAR(interp->importlib); Py_CLEAR(interp->import_func); @@ -308,6 +306,14 @@ PyInterpreterState_Clear(PyInterpreterState *interp) if (_PyRuntimeState_GetFinalizing(runtime) == NULL) { _PyWarnings_Fini(interp); } + /* We don't clear sysdict and builtins until the end of this function. + Because clearing other attributes can execute arbitrary Python code + which requires sysdict and builtins. */ + PyDict_Clear(interp->sysdict); + PyDict_Clear(interp->builtins); + Py_CLEAR(interp->sysdict); + Py_CLEAR(interp->builtins); + // XXX Once we have one allocator per interpreter (i.e. // per-interpreter GC) we must ensure that all of the interpreter's // objects have been cleaned up at the point. From webhook-mailer at python.org Thu Aug 13 04:48:54 2020 From: webhook-mailer at python.org (Pablo Galindo) Date: Thu, 13 Aug 2020 08:48:54 -0000 Subject: [Python-checkins] bpo-41531: Fix compilation of dict literals with more than 0xFFFF elements (GH-21850) Message-ID: https://github.com/python/cpython/commit/c51db0ea40ddabaf5f771ea633b37fcf4c90a495 commit: c51db0ea40ddabaf5f771ea633b37fcf4c90a495 branch: master author: Pablo Galindo committer: GitHub date: 2020-08-13T09:48:41+01:00 summary: bpo-41531: Fix compilation of dict literals with more than 0xFFFF elements (GH-21850) files: A Misc/NEWS.d/next/Core and Builtins/2020-08-12-19-32-15.bpo-41531.WgPzjT.rst M Lib/test/test_compile.py M Python/compile.c diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index 3dd8c8d1db810..6055192bf7050 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -752,6 +752,16 @@ def continue_in_while(): self.assertEqual(None, opcodes[0].argval) self.assertEqual('RETURN_VALUE', opcodes[1].opname) + def test_big_dict_literal(self): + # The compiler has a flushing point in "compiler_dict" that calls compiles + # a portion of the dictionary literal when the loop that iterates over the items + # reaches 0xFFFF elements but the code was not including the boundary element, + # dropping the key at position 0xFFFF. See bpo-41531 for more information + + dict_size = 0xFFFF + 1 + the_dict = "{" + ",".join(f"{x}:{x}" for x in range(dict_size)) + "}" + self.assertEqual(len(eval(the_dict)), dict_size) + class TestExpressionStackSize(unittest.TestCase): # These tests check that the computed stack size for a code object # stays within reasonable bounds (see issue #21523 for an example diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-12-19-32-15.bpo-41531.WgPzjT.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-12-19-32-15.bpo-41531.WgPzjT.rst new file mode 100644 index 0000000000000..8544664f39335 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-12-19-32-15.bpo-41531.WgPzjT.rst @@ -0,0 +1,2 @@ +Fix a bug that was dropping keys when compiling dict literals with more than +0xFFFF elements. Patch by Pablo Galindo. diff --git a/Python/compile.c b/Python/compile.c index 5dbd9f221fdf1..2c5326686f866 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -3894,7 +3894,7 @@ compiler_dict(struct compiler *c, expr_ty e) } else { if (elements == 0xFFFF) { - if (!compiler_subdict(c, e, i - elements, i)) { + if (!compiler_subdict(c, e, i - elements, i + 1)) { return 0; } if (have_dict) { From webhook-mailer at python.org Thu Aug 13 10:23:19 2020 From: webhook-mailer at python.org (Mohamed Koubaa) Date: Thu, 13 Aug 2020 14:23:19 -0000 Subject: [Python-checkins] bpo-1635741: Port _winapi ext to multi-stage init (GH-21371) Message-ID: https://github.com/python/cpython/commit/e087f7cd43dfa4223c55a8ecd71f4a7d685178e4 commit: e087f7cd43dfa4223c55a8ecd71f4a7d685178e4 branch: master author: Mohamed Koubaa committer: GitHub date: 2020-08-13T16:22:48+02:00 summary: bpo-1635741: Port _winapi ext to multi-stage init (GH-21371) files: A Misc/NEWS.d/next/Core and Builtins/2020-07-06-20-43-19.bpo-1635741.LYhsni.rst M Modules/_winapi.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-07-06-20-43-19.bpo-1635741.LYhsni.rst b/Misc/NEWS.d/next/Core and Builtins/2020-07-06-20-43-19.bpo-1635741.LYhsni.rst new file mode 100644 index 0000000000000..956fcd5d1ee29 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-07-06-20-43-19.bpo-1635741.LYhsni.rst @@ -0,0 +1 @@ +Port :mod:`winapi` to multiphase initialization diff --git a/Modules/_winapi.c b/Modules/_winapi.c index ddb11aa5a8204..7ba14095c96e1 100644 --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -35,8 +35,10 @@ /* See http://www.python.org/2.4/license for licensing details. */ #include "Python.h" +#include "moduleobject.h" // PyModuleDef_Slot #include "structmember.h" // PyMemberDef + #define WINDOWS_LEAN_AND_MEAN #include "windows.h" #include @@ -78,6 +80,17 @@ check_CancelIoEx() return has_CancelIoEx; } +typedef struct { + PyTypeObject *overlapped_type; +} WinApiState; + +static inline WinApiState* +winapi_get_state(PyObject *module) +{ + void *state = PyModule_GetState(module); + assert(state != NULL); + return (WinApiState *)state; +} /* * A Python object wrapping an OVERLAPPED structure and other useful data @@ -140,7 +153,9 @@ overlapped_dealloc(OverlappedObject *self) if (self->write_buffer.obj) PyBuffer_Release(&self->write_buffer); Py_CLEAR(self->read_buffer); - PyObject_Del(self); + PyTypeObject *tp = Py_TYPE(self); + tp->tp_free(self); + Py_DECREF(tp); } /*[clinic input] @@ -305,55 +320,29 @@ static PyMemberDef overlapped_members[] = { {NULL} }; -PyTypeObject OverlappedType = { - PyVarObject_HEAD_INIT(NULL, 0) - /* tp_name */ "_winapi.Overlapped", - /* tp_basicsize */ sizeof(OverlappedObject), - /* tp_itemsize */ 0, - /* tp_dealloc */ (destructor) overlapped_dealloc, - /* tp_vectorcall_offset */ 0, - /* tp_getattr */ 0, - /* tp_setattr */ 0, - /* tp_as_async */ 0, - /* tp_repr */ 0, - /* tp_as_number */ 0, - /* tp_as_sequence */ 0, - /* tp_as_mapping */ 0, - /* tp_hash */ 0, - /* tp_call */ 0, - /* tp_str */ 0, - /* tp_getattro */ 0, - /* tp_setattro */ 0, - /* tp_as_buffer */ 0, - /* tp_flags */ Py_TPFLAGS_DEFAULT, - /* tp_doc */ "OVERLAPPED structure wrapper", - /* tp_traverse */ 0, - /* tp_clear */ 0, - /* tp_richcompare */ 0, - /* tp_weaklistoffset */ 0, - /* tp_iter */ 0, - /* tp_iternext */ 0, - /* tp_methods */ overlapped_methods, - /* tp_members */ overlapped_members, - /* tp_getset */ 0, - /* tp_base */ 0, - /* tp_dict */ 0, - /* tp_descr_get */ 0, - /* tp_descr_set */ 0, - /* tp_dictoffset */ 0, - /* tp_init */ 0, - /* tp_alloc */ 0, - /* tp_new */ 0, +static PyType_Slot winapi_overlapped_type_slots[] = { + {Py_tp_dealloc, overlapped_dealloc}, + {Py_tp_doc, "OVERLAPPED structure wrapper"}, + {Py_tp_methods, overlapped_methods}, + {Py_tp_members, overlapped_members}, + {0,0} +}; + +static PyType_Spec winapi_overlapped_type_spec = { + .name = "_winapi.Overlapped", + .basicsize = sizeof(OverlappedObject), + .flags = Py_TPFLAGS_DEFAULT, + .slots = winapi_overlapped_type_slots, }; static OverlappedObject * -new_overlapped(HANDLE handle) +new_overlapped(PyObject *module, HANDLE handle) { - OverlappedObject *self; - - self = PyObject_New(OverlappedObject, &OverlappedType); + WinApiState *st = winapi_get_state(module); + OverlappedObject *self = PyObject_New(OverlappedObject, st->overlapped_type); if (!self) return NULL; + self->handle = handle; self->read_buffer = NULL; self->pending = 0; @@ -409,7 +398,7 @@ _winapi_ConnectNamedPipe_impl(PyObject *module, HANDLE handle, OverlappedObject *overlapped = NULL; if (use_overlapped) { - overlapped = new_overlapped(handle); + overlapped = new_overlapped(module, handle); if (!overlapped) return NULL; } @@ -1527,7 +1516,7 @@ _winapi_ReadFile_impl(PyObject *module, HANDLE handle, DWORD size, if (!buf) return NULL; if (use_overlapped) { - overlapped = new_overlapped(handle); + overlapped = new_overlapped(module, handle); if (!overlapped) { Py_DECREF(buf); return NULL; @@ -1810,7 +1799,7 @@ _winapi_WriteFile_impl(PyObject *module, HANDLE handle, PyObject *buffer, OverlappedObject *overlapped = NULL; if (use_overlapped) { - overlapped = new_overlapped(handle); + overlapped = new_overlapped(module, handle); if (!overlapped) return NULL; buf = &overlapped->write_buffer; @@ -1921,36 +1910,33 @@ static PyMethodDef winapi_functions[] = { {NULL, NULL} }; -static struct PyModuleDef winapi_module = { - PyModuleDef_HEAD_INIT, - "_winapi", - NULL, - -1, - winapi_functions, - NULL, - NULL, - NULL, - NULL -}; - #define WINAPI_CONSTANT(fmt, con) \ - PyDict_SetItemString(d, #con, Py_BuildValue(fmt, con)) - -PyMODINIT_FUNC -PyInit__winapi(void) + do { \ + PyObject *value = Py_BuildValue(fmt, con); \ + if (value == NULL) { \ + return -1; \ + } \ + if (PyDict_SetItemString(d, #con, value) < 0) { \ + Py_DECREF(value); \ + return -1; \ + } \ + Py_DECREF(value); \ + } while (0) + +static int winapi_exec(PyObject *m) { - PyObject *d; - PyObject *m; + WinApiState *st = winapi_get_state(m); - if (PyType_Ready(&OverlappedType) < 0) - return NULL; + st->overlapped_type = (PyTypeObject *)PyType_FromModuleAndSpec(m, &winapi_overlapped_type_spec, NULL); + if (st->overlapped_type == NULL) { + return -1; + } - m = PyModule_Create(&winapi_module); - if (m == NULL) - return NULL; - d = PyModule_GetDict(m); + if (PyModule_AddType(m, st->overlapped_type) < 0) { + return -1; + } - PyDict_SetItemString(d, "Overlapped", (PyObject *) &OverlappedType); + PyObject *d = PyModule_GetDict(m); /* constants */ WINAPI_CONSTANT(F_DWORD, CREATE_NEW_CONSOLE); @@ -2049,5 +2035,24 @@ PyInit__winapi(void) WINAPI_CONSTANT("i", NULL); - return m; + return 0; +} + +static PyModuleDef_Slot winapi_slots[] = { + {Py_mod_exec, winapi_exec}, + {0, NULL} +}; + +static struct PyModuleDef winapi_module = { + PyModuleDef_HEAD_INIT, + .m_name = "_winapi", + .m_size = sizeof(WinApiState), + .m_methods = winapi_functions, + .m_slots = winapi_slots, +}; + +PyMODINIT_FUNC +PyInit__winapi(void) +{ + return PyModuleDef_Init(&winapi_module); } From webhook-mailer at python.org Thu Aug 13 13:15:43 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Thu, 13 Aug 2020 17:15:43 -0000 Subject: [Python-checkins] bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst (GH-21858) Message-ID: https://github.com/python/cpython/commit/43577c01a2ab49122db696e9eaec6cb31d11cc81 commit: 43577c01a2ab49122db696e9eaec6cb31d11cc81 branch: master author: Victor Stinner committer: GitHub date: 2020-08-13T19:15:38+02:00 summary: bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst (GH-21858) Use generic '.. object::' to declare markers, rather than abusing '.. c:function::' which fails on Sphinx 3. files: M Doc/howto/instrumentation.rst diff --git a/Doc/howto/instrumentation.rst b/Doc/howto/instrumentation.rst index 909deb5fed33f..f0081e4ec2890 100644 --- a/Doc/howto/instrumentation.rst +++ b/Doc/howto/instrumentation.rst @@ -272,9 +272,7 @@ should instead read: Available static markers ------------------------ -.. I'm reusing the "c:function" type for markers - -.. c:function:: function__entry(str filename, str funcname, int lineno) +.. object:: function__entry(str filename, str funcname, int lineno) This marker indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions. @@ -290,7 +288,7 @@ Available static markers * ``$arg3`` : ``int`` line number -.. c:function:: function__return(str filename, str funcname, int lineno) +.. object:: function__return(str filename, str funcname, int lineno) This marker is the converse of :c:func:`function__entry`, and indicates that execution of a Python function has ended (either via ``return``, or via an @@ -298,7 +296,7 @@ Available static markers The arguments are the same as for :c:func:`function__entry` -.. c:function:: line(str filename, str funcname, int lineno) +.. object:: line(str filename, str funcname, int lineno) This marker indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. It is @@ -306,24 +304,24 @@ Available static markers The arguments are the same as for :c:func:`function__entry`. -.. c:function:: gc__start(int generation) +.. object:: gc__start(int generation) Fires when the Python interpreter starts a garbage collection cycle. ``arg0`` is the generation to scan, like :func:`gc.collect()`. -.. c:function:: gc__done(long collected) +.. object:: gc__done(long collected) Fires when the Python interpreter finishes a garbage collection cycle. ``arg0`` is the number of collected objects. -.. c:function:: import__find__load__start(str modulename) +.. object:: import__find__load__start(str modulename) Fires before :mod:`importlib` attempts to find and load the module. ``arg0`` is the module name. .. versionadded:: 3.7 -.. c:function:: import__find__load__done(str modulename, int found) +.. object:: import__find__load__done(str modulename, int found) Fires after :mod:`importlib`'s find_and_load function is called. ``arg0`` is the module name, ``arg1`` indicates if module was @@ -332,7 +330,7 @@ Available static markers .. versionadded:: 3.7 -.. c:function:: audit(str event, void *tuple) +.. object:: audit(str event, void *tuple) Fires when :func:`sys.audit` or :c:func:`PySys_Audit` is called. ``arg0`` is the event name as C string, ``arg1`` is a :c:type:`PyObject` @@ -375,14 +373,14 @@ If this file is installed in SystemTap's tapset directory (e.g. ``/usr/share/systemtap/tapset``), then these additional probepoints become available: -.. c:function:: python.function.entry(str filename, str funcname, int lineno, frameptr) +.. object:: python.function.entry(str filename, str funcname, int lineno, frameptr) This probe point indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions. -.. c:function:: python.function.return(str filename, str funcname, int lineno, frameptr) +.. object:: python.function.return(str filename, str funcname, int lineno, frameptr) - This probe point is the converse of :c:func:`python.function.return`, and + This probe point is the converse of ``python.function.return``, and indicates that execution of a Python function has ended (either via ``return``, or via an exception). It is only triggered for pure-Python (bytecode) functions. From webhook-mailer at python.org Thu Aug 13 13:16:06 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Thu, 13 Aug 2020 17:16:06 -0000 Subject: [Python-checkins] bpo-40204: Fix duplicates in the documentation (GH-21857) Message-ID: https://github.com/python/cpython/commit/46d10b1237c67ff8347f533eda6a5468d098f7eb commit: 46d10b1237c67ff8347f533eda6a5468d098f7eb branch: master author: Victor Stinner committer: GitHub date: 2020-08-13T19:16:02+02:00 summary: bpo-40204: Fix duplicates in the documentation (GH-21857) Fix two Sphinx 3 issues: Doc/c-api/buffer.rst:304: WARNING: Duplicate C declaration, also defined in 'c-api/buffer'. Declaration is 'PyBUF_ND'. Doc/c-api/unicode.rst:1603: WARNING: Duplicate C declaration, also defined in 'c-api/unicode'. Declaration is 'PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors)'. files: M Doc/c-api/buffer.rst M Doc/c-api/unicode.rst diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index fc1430efa8a4b..ddb28af88980c 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -301,7 +301,7 @@ must be C-contiguous. +-----------------------------------+-------+---------+------------+--------+ | .. c:macro:: PyBUF_ANY_CONTIGUOUS | yes | yes | NULL | C or F | +-----------------------------------+-------+---------+------------+--------+ -| .. c:macro:: PyBUF_ND | yes | NULL | NULL | C | +| :c:macro:`PyBUF_ND` | yes | NULL | NULL | C | +-----------------------------------+-------+---------+------------+--------+ diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index f3f0c4c6c2b9b..577cdf2511466 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -1487,17 +1487,21 @@ These are the mapping codec APIs: The following codec API is special in that maps Unicode to Unicode. -.. c:function:: PyObject* PyUnicode_Translate(PyObject *unicode, \ - PyObject *mapping, const char *errors) +.. c:function:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors) - Translate a Unicode object using the given *mapping* object and return the - resulting Unicode object. Return ``NULL`` if an exception was raised by the + Translate a string by applying a character mapping table to it and return the + resulting Unicode object. Return ``NULL`` if an exception was raised by the codec. - The *mapping* object must map Unicode ordinal integers to Unicode strings, - integers (which are then interpreted as Unicode ordinals) or ``None`` - (causing deletion of the character). Unmapped character ordinals (ones - which cause a :exc:`LookupError`) are left untouched and are copied as-is. + The mapping table must map Unicode ordinal integers to Unicode ordinal integers + or ``None`` (causing deletion of the character). + + Mapping tables need only provide the :meth:`__getitem__` interface; dictionaries + and sequences work well. Unmapped character ordinals (ones which cause a + :exc:`LookupError`) are left untouched and are copied as-is. + + *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to + use the default error handling. .. c:function:: PyObject* PyUnicode_TranslateCharmap(const Py_UNICODE *s, Py_ssize_t size, \ @@ -1600,23 +1604,6 @@ They all return ``NULL`` or ``-1`` if an exception occurs. characters are not included in the resulting strings. -.. c:function:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, \ - const char *errors) - - Translate a string by applying a character mapping table to it and return the - resulting Unicode object. - - The mapping table must map Unicode ordinal integers to Unicode ordinal integers - or ``None`` (causing deletion of the character). - - Mapping tables need only provide the :meth:`__getitem__` interface; dictionaries - and sequences work well. Unmapped character ordinals (ones which cause a - :exc:`LookupError`) are left untouched and are copied as-is. - - *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to - use the default error handling. - - .. c:function:: PyObject* PyUnicode_Join(PyObject *separator, PyObject *seq) Join a sequence of strings using the given *separator* and return the resulting From webhook-mailer at python.org Thu Aug 13 13:18:58 2020 From: webhook-mailer at python.org (Terry Jan Reedy) Date: Thu, 13 Aug 2020 17:18:58 -0000 Subject: [Python-checkins] bpo-41520: Fix second codeop regression (GH-21848) Message-ID: https://github.com/python/cpython/commit/c818b15fa59039de67022c29085d439fa5d3ef95 commit: c818b15fa59039de67022c29085d439fa5d3ef95 branch: master author: Terry Jan Reedy committer: GitHub date: 2020-08-13T13:18:49-04:00 summary: bpo-41520: Fix second codeop regression (GH-21848) * bpo-41520: Fix second codeop repression Fix the repression introduced by the initial regression fix. files: M Lib/codeop.py M Lib/test/test_codeop.py M Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst diff --git a/Lib/codeop.py b/Lib/codeop.py index 547629262d068..4c10470aee7b7 100644 --- a/Lib/codeop.py +++ b/Lib/codeop.py @@ -85,9 +85,9 @@ def _maybe_compile(compiler, source, filename, symbol): pass # Catch syntax warnings after the first compile - # to emit SyntaxWarning at most once. + # to emit warnings (SyntaxWarning, DeprecationWarning) at most once. with warnings.catch_warnings(): - warnings.simplefilter("error", SyntaxWarning) + warnings.simplefilter("error") try: code1 = compiler(source + "\n", filename, symbol) diff --git a/Lib/test/test_codeop.py b/Lib/test/test_codeop.py index 7984e5f1e5a69..45d0a7de9d925 100644 --- a/Lib/test/test_codeop.py +++ b/Lib/test/test_codeop.py @@ -307,14 +307,17 @@ def test_filename(self): def test_warning(self): # Test that the warning is only returned once. - with warnings_helper.check_warnings((".*literal", SyntaxWarning)) as w: - compile_command("0 is 0") - self.assertEqual(len(w.warnings), 1) + with warnings_helper.check_warnings( + (".*literal", SyntaxWarning), + (".*invalid", DeprecationWarning), + ) as w: + compile_command(r"'\e' is 0") + self.assertEqual(len(w.warnings), 2) # bpo-41520: check SyntaxWarning treated as an SyntaxError - with self.assertRaises(SyntaxError): + with warnings.catch_warnings(), self.assertRaises(SyntaxError): warnings.simplefilter('error', SyntaxWarning) - compile_command('1 is 1\n', symbol='exec') + compile_command('1 is 1', symbol='exec') if __name__ == "__main__": diff --git a/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst index ca5501c2aeec0..0e140d91bb4b6 100644 --- a/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst +++ b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst @@ -1 +1 @@ -Fix :mod:`codeop` regression: it no longer ignores :exc:`SyntaxWarning`. +Fix :mod:`codeop` regression that prevented turning compile warnings into errors. From webhook-mailer at python.org Thu Aug 13 13:20:36 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Thu, 13 Aug 2020 17:20:36 -0000 Subject: [Python-checkins] bpo-41521: Replace denylist with blocklist is http.cookiejar doc (GH-21826) Message-ID: https://github.com/python/cpython/commit/20ae565bd2d79425d5567c001ed8f89848d7d907 commit: 20ae565bd2d79425d5567c001ed8f89848d7d907 branch: master author: Victor Stinner committer: GitHub date: 2020-08-13T19:20:28+02:00 summary: bpo-41521: Replace denylist with blocklist is http.cookiejar doc (GH-21826) The http.cookiejar module has is_blocked() and blocked_domains() methods, so "blocklist" term sounds better than "denylist" in this module. Replace also denylisted with denied in test___all__. files: M Doc/library/http.cookiejar.rst M Lib/test/test___all__.py diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 0a9ee7eb516f4..7b1aa80a01797 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -462,8 +462,8 @@ receiving cookies. There are also some strictness switches that allow you to tighten up the rather loose Netscape protocol rules a little bit (at the cost of blocking some benign cookies). -A domain denylist and allowlist is provided (both off by default). Only domains -not in the denylist and present in the allowlist (if the allowlist is active) +A domain blocklist and allowlist is provided (both off by default). Only domains +not in the blocklist and present in the allowlist (if the allowlist is active) participate in cookie setting and returning. Use the *blocked_domains* constructor argument, and :meth:`blocked_domains` and :meth:`set_blocked_domains` methods (and the corresponding argument and methods @@ -471,7 +471,7 @@ for *allowed_domains*). If you set an allowlist, you can turn it off again by setting it to :const:`None`. Domains in block or allow lists that do not start with a dot must equal the -cookie domain to be matched. For example, ``"example.com"`` matches a denylist +cookie domain to be matched. For example, ``"example.com"`` matches a blocklist entry of ``"example.com"``, but ``"www.example.com"`` does not. Domains that do start with a dot are matched by more specific domains too. For example, both ``"www.example.com"`` and ``"www.coyote.example.com"`` match ``".example.com"`` @@ -494,7 +494,7 @@ and ``".168.1.2"``, 192.168.1.2 is blocked, but 193.168.1.2 is not. .. method:: DefaultCookiePolicy.is_blocked(domain) - Return whether *domain* is on the denylist for setting or receiving cookies. + Return whether *domain* is on the blocklist for setting or receiving cookies. .. method:: DefaultCookiePolicy.allowed_domains() diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py index c6ce64864c0db..15f42d2d114a6 100644 --- a/Lib/test/test___all__.py +++ b/Lib/test/test___all__.py @@ -85,13 +85,13 @@ def test_all(self): lib_dir = os.path.dirname(os.path.dirname(__file__)) for path, modname in self.walk_modules(lib_dir, ""): m = modname - denylisted = False + denied = False while m: if m in denylist: - denylisted = True + denied = True break m = m.rpartition('.')[0] - if denylisted: + if denied: continue if support.verbose: print(modname) From webhook-mailer at python.org Thu Aug 13 14:39:04 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Thu, 13 Aug 2020 18:39:04 -0000 Subject: [Python-checkins] [3.9] bpo-41520: Fix second codeop regression (GH-21848) Message-ID: https://github.com/python/cpython/commit/a3416c13b51af25675e175d4ffe07d8b925e7dbf commit: a3416c13b51af25675e175d4ffe07d8b925e7dbf branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-13T11:38:55-07:00 summary: [3.9] bpo-41520: Fix second codeop regression (GH-21848) Fix the repression introduced by the initial regression fix. (cherry picked from commit c818b15fa59039de67022c29085d439fa5d3ef95) Co-authored-by: Terry Jan Reedy (cherry picked from commit f24430f1542ea2768793b48704ae2d4e241892ae) Co-authored-by: Terry Jan Reedy files: M Lib/codeop.py M Lib/test/test_codeop.py M Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst diff --git a/Lib/codeop.py b/Lib/codeop.py index 97043877d1869..04ca6b9317087 100644 --- a/Lib/codeop.py +++ b/Lib/codeop.py @@ -85,9 +85,9 @@ def _maybe_compile(compiler, source, filename, symbol): pass # Catch syntax warnings after the first compile - # to emit SyntaxWarning at most once. + # to emit warnings (SyntaxWarning, DeprecationWarning) at most once. with warnings.catch_warnings(): - warnings.simplefilter("error", SyntaxWarning) + warnings.simplefilter("error") try: code1 = compiler(source + "\n", filename, symbol) diff --git a/Lib/test/test_codeop.py b/Lib/test/test_codeop.py index 1eae26b6ce17e..01ee00c3b4c91 100644 --- a/Lib/test/test_codeop.py +++ b/Lib/test/test_codeop.py @@ -297,14 +297,17 @@ def test_filename(self): def test_warning(self): # Test that the warning is only returned once. - with support.check_warnings((".*literal", SyntaxWarning)) as w: - compile_command("0 is 0") - self.assertEqual(len(w.warnings), 1) + with support.check_warnings( + (".*literal", SyntaxWarning), + (".*invalid", DeprecationWarning), + ) as w: + compile_command(r"'\e' is 0") + self.assertEqual(len(w.warnings), 2) # bpo-41520: check SyntaxWarning treated as an SyntaxError - with self.assertRaises(SyntaxError): + with warnings.catch_warnings(), self.assertRaises(SyntaxError): warnings.simplefilter('error', SyntaxWarning) - compile_command('1 is 1\n', symbol='exec') + compile_command('1 is 1', symbol='exec') if __name__ == "__main__": diff --git a/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst index ca5501c2aeec0..0e140d91bb4b6 100644 --- a/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst +++ b/Misc/NEWS.d/next/Library/2020-08-12-13-25-16.bpo-41520.BEUWa4.rst @@ -1 +1 @@ -Fix :mod:`codeop` regression: it no longer ignores :exc:`SyntaxWarning`. +Fix :mod:`codeop` regression that prevented turning compile warnings into errors. From webhook-mailer at python.org Thu Aug 13 15:42:02 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Thu, 13 Aug 2020 19:42:02 -0000 Subject: [Python-checkins] bpo-40204: Add :noindex: in the documentation (GH-21859) Message-ID: https://github.com/python/cpython/commit/d3ded080482beae578faa704b13534a62d066f9f commit: d3ded080482beae578faa704b13534a62d066f9f branch: master author: Victor Stinner committer: GitHub date: 2020-08-13T21:41:54+02:00 summary: bpo-40204: Add :noindex: in the documentation (GH-21859) Add :noindex: to duplicated documentation to fix "duplicate object description" errors. For example, fix this Sphinx 3 issue: Doc/library/configparser.rst:1146: WARNING: duplicate object description of configparser.ConfigParser.optionxform, other instance in library/configparser, use :noindex: for one of them files: M Doc/library/aifc.rst M Doc/library/configparser.rst M Doc/library/difflib.rst M Doc/library/enum.rst M Doc/library/socket.rst M Doc/library/tarfile.rst M Doc/library/token.rst M Doc/library/turtle.rst M Doc/library/urllib.request.rst diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst index 7328907730fb1..2e917cf7321b8 100644 --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -208,6 +208,7 @@ number of frames must be filled in. .. method:: aifc.tell() + :noindex: Return the current write position in the output file. Useful in combination with :meth:`setmark`. @@ -232,6 +233,7 @@ number of frames must be filled in. .. method:: aifc.close() + :noindex: Close the AIFF file. The header of the file is updated to reflect the actual size of the audio data. After calling this method, the object can no longer be diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 739477f55fddd..2e22a549ee281 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -674,97 +674,98 @@ be overridden by subclasses or by attribute assignment. .. attribute:: ConfigParser.BOOLEAN_STATES - By default when using :meth:`~ConfigParser.getboolean`, config parsers - consider the following values ``True``: ``'1'``, ``'yes'``, ``'true'``, - ``'on'`` and the following values ``False``: ``'0'``, ``'no'``, ``'false'``, - ``'off'``. You can override this by specifying a custom dictionary of strings - and their Boolean outcomes. For example: - - .. doctest:: - - >>> custom = configparser.ConfigParser() - >>> custom['section1'] = {'funky': 'nope'} - >>> custom['section1'].getboolean('funky') - Traceback (most recent call last): - ... - ValueError: Not a boolean: nope - >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} - >>> custom['section1'].getboolean('funky') - False - - Other typical Boolean pairs include ``accept``/``reject`` or - ``enabled``/``disabled``. + By default when using :meth:`~ConfigParser.getboolean`, config parsers + consider the following values ``True``: ``'1'``, ``'yes'``, ``'true'``, + ``'on'`` and the following values ``False``: ``'0'``, ``'no'``, ``'false'``, + ``'off'``. You can override this by specifying a custom dictionary of strings + and their Boolean outcomes. For example: + + .. doctest:: + + >>> custom = configparser.ConfigParser() + >>> custom['section1'] = {'funky': 'nope'} + >>> custom['section1'].getboolean('funky') + Traceback (most recent call last): + ... + ValueError: Not a boolean: nope + >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} + >>> custom['section1'].getboolean('funky') + False + + Other typical Boolean pairs include ``accept``/``reject`` or + ``enabled``/``disabled``. .. method:: ConfigParser.optionxform(option) + :noindex: - This method transforms option names on every read, get, or set - operation. The default converts the name to lowercase. This also - means that when a configuration file gets written, all keys will be - lowercase. Override this method if that's unsuitable. - For example: + This method transforms option names on every read, get, or set + operation. The default converts the name to lowercase. This also + means that when a configuration file gets written, all keys will be + lowercase. Override this method if that's unsuitable. + For example: - .. doctest:: + .. doctest:: + + >>> config = """ + ... [Section1] + ... Key = Value + ... + ... [Section2] + ... AnotherKey = Value + ... """ + >>> typical = configparser.ConfigParser() + >>> typical.read_string(config) + >>> list(typical['Section1'].keys()) + ['key'] + >>> list(typical['Section2'].keys()) + ['anotherkey'] + >>> custom = configparser.RawConfigParser() + >>> custom.optionxform = lambda option: option + >>> custom.read_string(config) + >>> list(custom['Section1'].keys()) + ['Key'] + >>> list(custom['Section2'].keys()) + ['AnotherKey'] - >>> config = """ - ... [Section1] - ... Key = Value - ... - ... [Section2] - ... AnotherKey = Value - ... """ - >>> typical = configparser.ConfigParser() - >>> typical.read_string(config) - >>> list(typical['Section1'].keys()) - ['key'] - >>> list(typical['Section2'].keys()) - ['anotherkey'] - >>> custom = configparser.RawConfigParser() - >>> custom.optionxform = lambda option: option - >>> custom.read_string(config) - >>> list(custom['Section1'].keys()) - ['Key'] - >>> list(custom['Section2'].keys()) - ['AnotherKey'] - - .. note:: - The optionxform function transforms option names to a canonical form. - This should be an idempotent function: if the name is already in - canonical form, it should be returned unchanged. + .. note:: + The optionxform function transforms option names to a canonical form. + This should be an idempotent function: if the name is already in + canonical form, it should be returned unchanged. .. attribute:: ConfigParser.SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered - part of the section name, thus ``[ larch ]`` will be read as a section of - name ``" larch "``. Override this attribute if that's unsuitable. For - example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: + + .. doctest:: + + >>> import re + >>> config = """ + ... [Section 1] + ... option = value + ... + ... [ Section 2 ] + ... another = val + ... """ + >>> typical = configparser.ConfigParser() + >>> typical.read_string(config) + >>> typical.sections() + ['Section 1', ' Section 2 '] + >>> custom = configparser.ConfigParser() + >>> custom.SECTCRE = re.compile(r"\[ *(?P
[^]]+?) *\]") + >>> custom.read_string(config) + >>> custom.sections() + ['Section 1', 'Section 2'] - .. doctest:: + .. note:: - >>> import re - >>> config = """ - ... [Section 1] - ... option = value - ... - ... [ Section 2 ] - ... another = val - ... """ - >>> typical = configparser.ConfigParser() - >>> typical.read_string(config) - >>> typical.sections() - ['Section 1', ' Section 2 '] - >>> custom = configparser.ConfigParser() - >>> custom.SECTCRE = re.compile(r"\[ *(?P
[^]]+?) *\]") - >>> custom.read_string(config) - >>> custom.sections() - ['Section 1', 'Section 2'] - - .. note:: - - While ConfigParser objects also use an ``OPTCRE`` attribute for recognizing - option lines, it's not recommended to override it because that would - interfere with constructor options *allow_no_value* and *delimiters*. + While ConfigParser objects also use an ``OPTCRE`` attribute for recognizing + option lines, it's not recommended to override it because that would + interfere with constructor options *allow_no_value* and *delimiters*. Legacy API Examples diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 7a898c21b52e0..25e3511d01785 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -24,6 +24,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. .. class:: SequenceMatcher + :noindex: This is a flexible class for comparing pairs of sequences of any type, so long as the sequence elements are :term:`hashable`. The basic algorithm predates, and is a @@ -651,6 +652,7 @@ The :class:`Differ` class has this constructor: .. class:: Differ(linejunk=None, charjunk=None) + :noindex: Optional keyword parameters *linejunk* and *charjunk* are for filter functions (or ``None``): diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index b327a0ad15f96..00bfb260c0204 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -50,6 +50,7 @@ helper, :class:`auto`. the bitwise operations without losing their :class:`Flag` membership. .. function:: unique + :noindex: Enum class decorator that ensures only one name is bound to any one value. diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index d798c1a9d10a0..5bcac20a4c604 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1695,7 +1695,9 @@ to sockets. .. method:: socket.setsockopt(level, optname, value: int) .. method:: socket.setsockopt(level, optname, value: buffer) + :noindex: .. method:: socket.setsockopt(level, optname, None, optlen: int) + :noindex: .. index:: module: struct diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index c204263d3a094..cca466b569794 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -151,6 +151,7 @@ Some facts and figures: .. class:: TarFile + :noindex: Class for reading and writing tar archives. Do not use this class directly: use :func:`tarfile.open` instead. See :ref:`tarfile-objects`. diff --git a/Doc/library/token.rst b/Doc/library/token.rst index f8ebb275ebbc4..a1aceba96ce03 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -70,6 +70,7 @@ the :mod:`tokenize` module. .. data:: TYPE_COMMENT + :noindex: Token value indicating that a type comment was recognized. Such tokens are only produced when :func:`ast.parse()` is invoked with diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index fed85045435b1..d3487537df99a 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1069,6 +1069,7 @@ More drawing control ~~~~~~~~~~~~~~~~~~~~ .. function:: reset() + :noindex: Delete the turtle's drawings from the screen, re-center the turtle and set variables to the default values. @@ -1090,6 +1091,7 @@ More drawing control .. function:: clear() + :noindex: Delete the turtle's drawings from the screen. Do not move turtle. State and position of the turtle as well as drawings of other turtles are not affected. @@ -1362,6 +1364,7 @@ Using events ------------ .. function:: onclick(fun, btn=1, add=None) + :noindex: :param fun: a function with two arguments which will be called with the coordinates of the clicked point on the canvas diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 288ce14d36f01..b37f230feb601 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -946,7 +946,7 @@ tracking URIs for which authentication credentials should always be sent. If *is_authenticated* is specified as ``True``, *realm* is ignored. -.. method:: HTTPPasswordMgr.find_user_password(realm, authuri) +.. method:: HTTPPasswordMgrWithPriorAuth.find_user_password(realm, authuri) Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects From webhook-mailer at python.org Thu Aug 13 15:52:14 2020 From: webhook-mailer at python.org (Srinivas Reddy Thatiparthy =?utf-8?q??= =?utf-8?b?KOCwtuCxjeCwsOCxgOCwqOCwv+CwteCwvuCwuOCxjSAg4LCw4LGG4LCh?= =?utf-8?b?4LGN4LCh4LC/IOCwpOCwvuCwn+Cwv+CwquCwsOCxjeCwpOCwvyk=?=) Date: Thu, 13 Aug 2020 19:52:14 -0000 Subject: [Python-checkins] bpo-41066: Update the comparison section for os vs pathlib (GH-21261) Message-ID: https://github.com/python/cpython/commit/0eb9deb4a62e6d9daa82bc2f67d1075864ca8ece commit: 0eb9deb4a62e6d9daa82bc2f67d1075864ca8ece branch: master author: Srinivas Reddy Thatiparthy (?????????? ?????? ?????????) committer: GitHub date: 2020-08-13T21:52:04+02:00 summary: bpo-41066: Update the comparison section for os vs pathlib (GH-21261) files: M Doc/library/pathlib.rst diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index bf6fee44df2c8..f705d15b8d4f1 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1196,9 +1196,12 @@ os and os.path pathlib :func:`os.path.exists` :meth:`Path.exists` :func:`os.path.expanduser` :meth:`Path.expanduser` and :meth:`Path.home` +:func:`os.listdir` :meth:`Path.iterdir` :func:`os.path.isdir` :meth:`Path.is_dir` :func:`os.path.isfile` :meth:`Path.is_file` :func:`os.path.islink` :meth:`Path.is_symlink` +:func:`os.link` :meth:`Path.link_to` +:func:`os.symlink` :meth:`Path.symlink_to` :func:`os.readlink` :meth:`Path.readlink` :func:`os.stat` :meth:`Path.stat`, :meth:`Path.owner`, From webhook-mailer at python.org Thu Aug 13 16:03:26 2020 From: webhook-mailer at python.org (Antoine Pitrou) Date: Thu, 13 Aug 2020 20:03:26 -0000 Subject: [Python-checkins] [3.8] bpo-41066: Update the comparison section for os vs pathlib (GH-21261) (GH-21864) Message-ID: https://github.com/python/cpython/commit/43b3e4c32976bf069241ad8bb174929a816ee7ba commit: 43b3e4c32976bf069241ad8bb174929a816ee7ba branch: 3.8 author: Antoine Pitrou committer: GitHub date: 2020-08-13T13:03:18-07:00 summary: [3.8] bpo-41066: Update the comparison section for os vs pathlib (GH-21261) (GH-21864) (cherry picked from commit 0eb9deb4a62e6d9daa82bc2f67d1075864ca8ece) Co-authored-by: Srinivas Reddy Thatiparthy (?????????? ?????? ?????????) Automerge-Triggered-By: @pitrou files: M Doc/library/pathlib.rst diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index b900d093b7565..6df352f0e3913 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1139,9 +1139,12 @@ os and os.path pathlib :func:`os.path.exists` :meth:`Path.exists` :func:`os.path.expanduser` :meth:`Path.expanduser` and :meth:`Path.home` +:func:`os.listdir` :meth:`Path.iterdir` :func:`os.path.isdir` :meth:`Path.is_dir` :func:`os.path.isfile` :meth:`Path.is_file` :func:`os.path.islink` :meth:`Path.is_symlink` +:func:`os.link` :meth:`Path.link_to` +:func:`os.symlink` :meth:`Path.symlink_to` :func:`os.stat` :meth:`Path.stat`, :meth:`Path.owner`, :meth:`Path.group` From webhook-mailer at python.org Thu Aug 13 16:11:59 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Thu, 13 Aug 2020 20:11:59 -0000 Subject: [Python-checkins] bpo-40204, doc: Fix syntax of C variables (GH-21846) Message-ID: https://github.com/python/cpython/commit/474652fe9346382dbf793f20b671eb74668bebde commit: 474652fe9346382dbf793f20b671eb74668bebde branch: master author: Victor Stinner committer: GitHub date: 2020-08-13T22:11:50+02:00 summary: bpo-40204, doc: Fix syntax of C variables (GH-21846) For example, fix the following Sphinx 3 errors: Doc/c-api/buffer.rst:102: WARNING: Error in declarator or parameters Invalid C declaration: Expected identifier in nested name. [error at 5] void \*obj -----^ Doc/c-api/arg.rst:130: WARNING: Unparseable C cross-reference: 'PyObject*' Invalid C declaration: Expected end of definition. [error at 8] PyObject* --------^ The modified documentation is compatible with Sphinx 2 and Sphinx 3. files: M Doc/c-api/arg.rst M Doc/c-api/buffer.rst M Doc/c-api/call.rst M Doc/c-api/capsule.rst M Doc/c-api/dict.rst M Doc/c-api/exceptions.rst M Doc/c-api/file.rst M Doc/c-api/init.rst M Doc/c-api/intro.rst M Doc/c-api/marshal.rst M Doc/c-api/memory.rst M Doc/c-api/module.rst M Doc/c-api/object.rst M Doc/c-api/structures.rst M Doc/c-api/tuple.rst M Doc/c-api/typeobj.rst M Doc/c-api/unicode.rst M Doc/c-api/veryhigh.rst M Doc/whatsnew/3.3.rst diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 26e872c5a348e..bdaae44e240a0 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -129,12 +129,12 @@ which disallows mutable objects such as :class:`bytearray`. ``S`` (:class:`bytes`) [PyBytesObject \*] Requires that the Python object is a :class:`bytes` object, without attempting any conversion. Raises :exc:`TypeError` if the object is not - a bytes object. The C variable may also be declared as :c:type:`PyObject\*`. + a bytes object. The C variable may also be declared as :c:type:`PyObject*`. ``Y`` (:class:`bytearray`) [PyByteArrayObject \*] Requires that the Python object is a :class:`bytearray` object, without attempting any conversion. Raises :exc:`TypeError` if the object is not - a :class:`bytearray` object. The C variable may also be declared as :c:type:`PyObject\*`. + a :class:`bytearray` object. The C variable may also be declared as :c:type:`PyObject*`. ``u`` (:class:`str`) [const Py_UNICODE \*] Convert a Python Unicode object to a C pointer to a NUL-terminated buffer of @@ -181,7 +181,7 @@ which disallows mutable objects such as :class:`bytearray`. ``U`` (:class:`str`) [PyObject \*] Requires that the Python object is a Unicode object, without attempting any conversion. Raises :exc:`TypeError` if the object is not a Unicode - object. The C variable may also be declared as :c:type:`PyObject\*`. + object. The C variable may also be declared as :c:type:`PyObject*`. ``w*`` (read-write :term:`bytes-like object`) [Py_buffer] This format accepts any object which implements the read-write buffer @@ -194,10 +194,10 @@ which disallows mutable objects such as :class:`bytearray`. It only works for encoded data without embedded NUL bytes. This format requires two arguments. The first is only used as input, and - must be a :c:type:`const char\*` which points to the name of an encoding as a + must be a :c:type:`const char*` which points to the name of an encoding as a NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used. An exception is raised if the named encoding is not known to Python. The - second argument must be a :c:type:`char\*\*`; the value of the pointer it + second argument must be a :c:type:`char**`; the value of the pointer it references will be set to a buffer with the contents of the argument text. The text will be encoded in the encoding specified by the first argument. @@ -217,10 +217,10 @@ which disallows mutable objects such as :class:`bytearray`. characters. It requires three arguments. The first is only used as input, and must be a - :c:type:`const char\*` which points to the name of an encoding as a + :c:type:`const char*` which points to the name of an encoding as a NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used. An exception is raised if the named encoding is not known to Python. The - second argument must be a :c:type:`char\*\*`; the value of the pointer it + second argument must be a :c:type:`char**`; the value of the pointer it references will be set to a buffer with the contents of the argument text. The text will be encoded in the encoding specified by the first argument. The third argument must be a pointer to an integer; the referenced integer @@ -320,7 +320,7 @@ Other objects ``O!`` (object) [*typeobject*, PyObject \*] Store a Python object in a C object pointer. This is similar to ``O``, but takes two C arguments: the first is the address of a Python type object, the - second is the address of the C variable (of type :c:type:`PyObject\*`) into which + second is the address of the C variable (of type :c:type:`PyObject*`) into which the object pointer is stored. If the Python object does not have the required type, :exc:`TypeError` is raised. @@ -329,13 +329,13 @@ Other objects ``O&`` (object) [*converter*, *anything*] Convert a Python object to a C variable through a *converter* function. This takes two arguments: the first is a function, the second is the address of a C - variable (of arbitrary type), converted to :c:type:`void \*`. The *converter* + variable (of arbitrary type), converted to :c:type:`void *`. The *converter* function in turn is called as follows:: status = converter(object, address); where *object* is the Python object to be converted and *address* is the - :c:type:`void\*` argument that was passed to the :c:func:`PyArg_Parse\*` function. + :c:type:`void*` argument that was passed to the :c:func:`PyArg_Parse\*` function. The returned *status* should be ``1`` for a successful conversion and ``0`` if the conversion has failed. When the conversion fails, the *converter* function should raise an exception and leave the content of *address* unmodified. @@ -481,7 +481,7 @@ API Functions *args*; it must actually be a tuple. The length of the tuple must be at least *min* and no more than *max*; *min* and *max* may be equal. Additional arguments must be passed to the function, each of which should be a pointer to a - :c:type:`PyObject\*` variable; these will be filled in with the values from + :c:type:`PyObject*` variable; these will be filled in with the values from *args*; they will contain borrowed references. The variables which correspond to optional parameters not given by *args* will not be filled in; these should be initialized by the caller. This function returns true on success and false if @@ -650,8 +650,8 @@ Building values ``O&`` (object) [*converter*, *anything*] Convert *anything* to a Python object through a *converter* function. The - function is called with *anything* (which should be compatible with :c:type:`void - \*`) as its argument and should return a "new" Python object, or ``NULL`` if an + function is called with *anything* (which should be compatible with :c:type:`void*`) + as its argument and should return a "new" Python object, or ``NULL`` if an error occurred. ``(items)`` (:class:`tuple`) [*matching-items*] diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index ddb28af88980c..e32719373cc71 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -89,7 +89,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. .. c:type:: Py_buffer - .. c:member:: void \*buf + .. c:member:: void *buf A pointer to the start of the logical structure described by the buffer fields. This can be any location within the underlying physical memory @@ -99,7 +99,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. For :term:`contiguous` arrays, the value points to the beginning of the memory block. - .. c:member:: void \*obj + .. c:member:: void *obj A new reference to the exporting object. The reference is owned by the consumer and automatically decremented and set to ``NULL`` by @@ -145,7 +145,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``. - .. c:member:: const char \*format + .. c:member:: const char *format A *NUL* terminated string in :mod:`struct` module style syntax describing the contents of a single item. If this is ``NULL``, ``"B"`` (unsigned bytes) @@ -164,7 +164,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. to 64. Exporters MUST respect this limit, consumers of multi-dimensional buffers SHOULD be able to handle up to :c:macro:`PyBUF_MAX_NDIM` dimensions. - .. c:member:: Py_ssize_t \*shape + .. c:member:: Py_ssize_t *shape An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` indicating the shape of the memory as an n-dimensional array. Note that @@ -177,7 +177,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. The shape array is read-only for the consumer. - .. c:member:: Py_ssize_t \*strides + .. c:member:: Py_ssize_t *strides An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` giving the number of bytes to skip to get to a new element in each @@ -189,7 +189,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. The strides array is read-only for the consumer. - .. c:member:: Py_ssize_t \*suboffsets + .. c:member:: Py_ssize_t *suboffsets An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`. If ``suboffsets[n] >= 0``, the values stored along the nth dimension are @@ -207,7 +207,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. The suboffsets array is read-only for the consumer. - .. c:member:: void \*internal + .. c:member:: void *internal This is for use internally by the exporting object. For example, this might be re-cast as an integer by the exporter and used to store flags @@ -438,12 +438,12 @@ Buffer-related functions Send a request to *exporter* to fill in *view* as specified by *flags*. If the exporter cannot provide a buffer of the exact type, it MUST raise - :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to ``NULL`` and + :c:data:`PyExc_BufferError`, set ``view->obj`` to ``NULL`` and return ``-1``. - On success, fill in *view*, set :c:member:`view->obj` to a new reference + On success, fill in *view*, set ``view->obj`` to a new reference to *exporter* and return 0. In the case of chained buffer providers - that redirect requests to a single object, :c:member:`view->obj` MAY + that redirect requests to a single object, ``view->obj`` MAY refer to this object instead of *exporter* (See :ref:`Buffer Object Structures `). Successful calls to :c:func:`PyObject_GetBuffer` must be paired with calls @@ -455,7 +455,7 @@ Buffer-related functions .. c:function:: void PyBuffer_Release(Py_buffer *view) Release the buffer *view* and decrement the reference count for - :c:member:`view->obj`. This function MUST be called when the buffer + ``view->obj``. This function MUST be called when the buffer is no longer being used, otherwise reference leaks may occur. It is an error to call this function on a buffer that was not obtained via @@ -516,9 +516,9 @@ Buffer-related functions *view* as specified by flags, unless *buf* has been designated as read-only and :c:macro:`PyBUF_WRITABLE` is set in *flags*. - On success, set :c:member:`view->obj` to a new reference to *exporter* and + On success, set ``view->obj`` to a new reference to *exporter* and return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set - :c:member:`view->obj` to ``NULL`` and return ``-1``; + ``view->obj`` to ``NULL`` and return ``-1``; If this function is used as part of a :ref:`getbufferproc `, *exporter* MUST be set to the exporting object and *flags* must be passed diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 0832e7e219360..31dc9c8031fdb 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -84,7 +84,7 @@ This is a pointer to a function with the following signature: and they must be unique. If there are no keyword arguments, then *kwnames* can instead be *NULL*. -.. c:var:: PY_VECTORCALL_ARGUMENTS_OFFSET +.. c:macro:: PY_VECTORCALL_ARGUMENTS_OFFSET If this flag is set in a vectorcall *nargsf* argument, the callee is allowed to temporarily change ``args[-1]``. In other words, *args* points to @@ -283,7 +283,7 @@ please see individual documentation for details. This is the equivalent of the Python expression: ``callable(*args)``. - Note that if you only pass :c:type:`PyObject \*` args, + Note that if you only pass :c:type:`PyObject *` args, :c:func:`PyObject_CallFunctionObjArgs` is a faster alternative. .. versionchanged:: 3.4 @@ -304,17 +304,17 @@ please see individual documentation for details. This is the equivalent of the Python expression: ``obj.name(arg1, arg2, ...)``. - Note that if you only pass :c:type:`PyObject \*` args, + Note that if you only pass :c:type:`PyObject *` args, :c:func:`PyObject_CallMethodObjArgs` is a faster alternative. .. versionchanged:: 3.4 The types of *name* and *format* were changed from ``char *``. -.. c:function:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ..., NULL) +.. c:function:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ...) Call a callable Python object *callable*, with a variable number of - :c:type:`PyObject \*` arguments. The arguments are provided as a variable number + :c:type:`PyObject *` arguments. The arguments are provided as a variable number of parameters followed by *NULL*. Return the result of the call on success, or raise an exception and return @@ -324,11 +324,11 @@ please see individual documentation for details. ``callable(arg1, arg2, ...)``. -.. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ..., NULL) +.. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ...) Call a method of the Python object *obj*, where the name of the method is given as a Python string object in *name*. It is called with a variable number of - :c:type:`PyObject \*` arguments. The arguments are provided as a variable number + :c:type:`PyObject *` arguments. The arguments are provided as a variable number of parameters followed by *NULL*. Return the result of the call on success, or raise an exception and return diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst index 78e21140b2f80..5eb313c89bfd5 100644 --- a/Doc/c-api/capsule.rst +++ b/Doc/c-api/capsule.rst @@ -15,7 +15,7 @@ Refer to :ref:`using-capsules` for more information on using these objects. .. c:type:: PyCapsule This subtype of :c:type:`PyObject` represents an opaque value, useful for C - extension modules who need to pass an opaque value (as a :c:type:`void\*` + extension modules who need to pass an opaque value (as a :c:type:`void*` pointer) through Python code to other C code. It is often used to make a C function pointer defined in one module available to other modules, so the regular import mechanism can be used to access C APIs defined in dynamically diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index 7493837ac622f..769484134ed51 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -73,7 +73,7 @@ Dictionary Objects .. index:: single: PyUnicode_FromString() Insert *val* into the dictionary *p* using *key* as a key. *key* should - be a :c:type:`const char\*`. The key object is created using + be a :c:type:`const char*`. The key object is created using ``PyUnicode_FromString(key)``. Return ``0`` on success or ``-1`` on failure. This function *does not* steal a reference to *val*. @@ -116,7 +116,7 @@ Dictionary Objects .. c:function:: PyObject* PyDict_GetItemString(PyObject *p, const char *key) This is the same as :c:func:`PyDict_GetItem`, but *key* is specified as a - :c:type:`const char\*`, rather than a :c:type:`PyObject\*`. + :c:type:`const char*`, rather than a :c:type:`PyObject*`. Note that exceptions which occur while calling :meth:`__hash__` and :meth:`__eq__` methods and creating a temporary string object @@ -165,7 +165,7 @@ Dictionary Objects prior to the first call to this function to start the iteration; the function returns true for each pair in the dictionary, and false once all pairs have been reported. The parameters *pkey* and *pvalue* should either - point to :c:type:`PyObject\*` variables that will be filled in with each key + point to :c:type:`PyObject*` variables that will be filled in with each key and value, respectively, or may be ``NULL``. Any references returned through them are borrowed. *ppos* should not be altered during iteration. Its value represents offsets within the internal dictionary structure, and diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index b4722ff81979f..247b6d68eceae 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -783,7 +783,7 @@ Standard Exceptions All standard Python exceptions are available as global variables whose names are ``PyExc_`` followed by the Python exception name. These have the type -:c:type:`PyObject\*`; they are all class objects. For completeness, here are all +:c:type:`PyObject*`; they are all class objects. For completeness, here are all the variables: .. index:: @@ -1003,7 +1003,7 @@ Standard Warning Categories All standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python exception name. These have the type -:c:type:`PyObject\*`; they are all class objects. For completeness, here are all +:c:type:`PyObject*`; they are all class objects. For completeness, here are all the variables: .. index:: diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst index 5370c4e350a0b..ea027ee975c65 100644 --- a/Doc/c-api/file.rst +++ b/Doc/c-api/file.rst @@ -8,7 +8,7 @@ File Objects .. index:: object: file These APIs are a minimal emulation of the Python 2 C API for built-in file -objects, which used to rely on the buffered I/O (:c:type:`FILE\*`) support +objects, which used to rely on the buffered I/O (:c:type:`FILE*`) support from the C standard library. In Python 3, files and streams use the new :mod:`io` module, which defines several layers over the low-level unbuffered I/O of the operating system. The functions described below are @@ -17,7 +17,7 @@ error reporting in the interpreter; third-party code is advised to access the :mod:`io` APIs instead. -.. c:function:: PyFile_FromFd(int fd, const char *name, const char *mode, int buffering, const char *encoding, const char *errors, const char *newline, int closefd) +.. c:function:: PyObject* PyFile_FromFd(int fd, const char *name, const char *mode, int buffering, const char *encoding, const char *errors, const char *newline, int closefd) Create a Python file object from the file descriptor of an already opened file *fd*. The arguments *name*, *encoding*, *errors* and *newline* diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 68fed2acc447e..7f06648bcb457 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -81,7 +81,7 @@ When a flag is set by an option, the value of the flag is the number of times that the option was set. For example, ``-b`` sets :c:data:`Py_BytesWarningFlag` to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. -.. c:var:: Py_BytesWarningFlag +.. c:var:: int Py_BytesWarningFlag Issue a warning when comparing :class:`bytes` or :class:`bytearray` with :class:`str` or :class:`bytes` with :class:`int`. Issue an error if greater @@ -89,7 +89,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-b` option. -.. c:var:: Py_DebugFlag +.. c:var:: int Py_DebugFlag Turn on parser debugging output (for expert only, depending on compilation options). @@ -97,7 +97,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-d` option and the :envvar:`PYTHONDEBUG` environment variable. -.. c:var:: Py_DontWriteBytecodeFlag +.. c:var:: int Py_DontWriteBytecodeFlag If set to non-zero, Python won't try to write ``.pyc`` files on the import of source modules. @@ -105,14 +105,14 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-B` option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable. -.. c:var:: Py_FrozenFlag +.. c:var:: int Py_FrozenFlag Suppress error messages when calculating the module search path in :c:func:`Py_GetPath`. Private flag used by ``_freeze_importlib`` and ``frozenmain`` programs. -.. c:var:: Py_HashRandomizationFlag +.. c:var:: int Py_HashRandomizationFlag Set to ``1`` if the :envvar:`PYTHONHASHSEED` environment variable is set to a non-empty string. @@ -120,14 +120,14 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. If the flag is non-zero, read the :envvar:`PYTHONHASHSEED` environment variable to initialize the secret hash seed. -.. c:var:: Py_IgnoreEnvironmentFlag +.. c:var:: int Py_IgnoreEnvironmentFlag Ignore all :envvar:`PYTHON*` environment variables, e.g. :envvar:`PYTHONPATH` and :envvar:`PYTHONHOME`, that might be set. Set by the :option:`-E` and :option:`-I` options. -.. c:var:: Py_InspectFlag +.. c:var:: int Py_InspectFlag When a script is passed as first argument or the :option:`-c` option is used, enter interactive mode after executing the script or the command, even when @@ -136,11 +136,11 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-i` option and the :envvar:`PYTHONINSPECT` environment variable. -.. c:var:: Py_InteractiveFlag +.. c:var:: int Py_InteractiveFlag Set by the :option:`-i` option. -.. c:var:: Py_IsolatedFlag +.. c:var:: int Py_IsolatedFlag Run Python in isolated mode. In isolated mode :data:`sys.path` contains neither the script's directory nor the user's site-packages directory. @@ -149,7 +149,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. versionadded:: 3.4 -.. c:var:: Py_LegacyWindowsFSEncodingFlag +.. c:var:: int Py_LegacyWindowsFSEncodingFlag If the flag is non-zero, use the ``mbcs`` encoding instead of the UTF-8 encoding for the filesystem encoding. @@ -161,7 +161,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. availability:: Windows. -.. c:var:: Py_LegacyWindowsStdioFlag +.. c:var:: int Py_LegacyWindowsStdioFlag If the flag is non-zero, use :class:`io.FileIO` instead of :class:`WindowsConsoleIO` for :mod:`sys` standard streams. @@ -173,7 +173,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. availability:: Windows. -.. c:var:: Py_NoSiteFlag +.. c:var:: int Py_NoSiteFlag Disable the import of the module :mod:`site` and the site-dependent manipulations of :data:`sys.path` that it entails. Also disable these @@ -182,7 +182,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-S` option. -.. c:var:: Py_NoUserSiteDirectory +.. c:var:: int Py_NoUserSiteDirectory Don't add the :data:`user site-packages directory ` to :data:`sys.path`. @@ -190,12 +190,12 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-s` and :option:`-I` options, and the :envvar:`PYTHONNOUSERSITE` environment variable. -.. c:var:: Py_OptimizeFlag +.. c:var:: int Py_OptimizeFlag Set by the :option:`-O` option and the :envvar:`PYTHONOPTIMIZE` environment variable. -.. c:var:: Py_QuietFlag +.. c:var:: int Py_QuietFlag Don't display the copyright and version messages even in interactive mode. @@ -203,14 +203,14 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. versionadded:: 3.2 -.. c:var:: Py_UnbufferedStdioFlag +.. c:var:: int Py_UnbufferedStdioFlag Force the stdout and stderr streams to be unbuffered. Set by the :option:`-u` option and the :envvar:`PYTHONUNBUFFERED` environment variable. -.. c:var:: Py_VerboseFlag +.. c:var:: int Py_VerboseFlag Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded. If greater or equal @@ -830,7 +830,7 @@ code, or when embedding the Python interpreter: .. c:type:: PyThreadState This data structure represents the state of a single thread. The only public - data member is :c:type:`PyInterpreterState \*`:attr:`interp`, which points to + data member is :attr:`interp` (:c:type:`PyInterpreterState *`), which points to this thread's interpreter state. @@ -1619,7 +1619,7 @@ The Python interpreter provides low-level support for thread-local storage (TLS) which wraps the underlying native TLS implementation to support the Python-level thread local storage API (:class:`threading.local`). The CPython C level APIs are similar to those offered by pthreads and Windows: -use a thread key and functions to associate a :c:type:`void\*` value per +use a thread key and functions to associate a :c:type:`void*` value per thread. The GIL does *not* need to be held when calling these functions; they supply @@ -1630,8 +1630,8 @@ you need to include :file:`pythread.h` to use thread-local storage. .. note:: None of these API functions handle memory management on behalf of the - :c:type:`void\*` values. You need to allocate and deallocate them yourself. - If the :c:type:`void\*` values happen to be :c:type:`PyObject\*`, these + :c:type:`void*` values. You need to allocate and deallocate them yourself. + If the :c:type:`void*` values happen to be :c:type:`PyObject*`, these functions don't do refcount operations on them either. .. _thread-specific-storage-api: @@ -1727,14 +1727,14 @@ undefined if the given :c:type:`Py_tss_t` has not been initialized by .. c:function:: int PyThread_tss_set(Py_tss_t *key, void *value) - Return a zero value to indicate successfully associating a :c:type:`void\*` + Return a zero value to indicate successfully associating a :c:type:`void*` value with a TSS key in the current thread. Each thread has a distinct - mapping of the key to a :c:type:`void\*` value. + mapping of the key to a :c:type:`void*` value. .. c:function:: void* PyThread_tss_get(Py_tss_t *key) - Return the :c:type:`void\*` value associated with a TSS key in the current + Return the :c:type:`void*` value associated with a TSS key in the current thread. This returns ``NULL`` if no value is associated with the key in the current thread. diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index e89a788de0d50..7ca8693afab79 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -229,13 +229,13 @@ Objects, Types and Reference Counts .. index:: object: type Most Python/C API functions have one or more arguments as well as a return value -of type :c:type:`PyObject\*`. This type is a pointer to an opaque data type +of type :c:type:`PyObject*`. This type is a pointer to an opaque data type representing an arbitrary Python object. Since all Python object types are treated the same way by the Python language in most situations (e.g., assignments, scope rules, and argument passing), it is only fitting that they should be represented by a single C type. Almost all Python objects live on the heap: you never declare an automatic or static variable of type -:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject\*` can be +:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject*` can be declared. The sole exception are the type objects; since these must never be deallocated, they are typically static :c:type:`PyTypeObject` objects. @@ -496,7 +496,7 @@ Types There are few other data types that play a significant role in the Python/C API; most are simple C types such as :c:type:`int`, :c:type:`long`, -:c:type:`double` and :c:type:`char\*`. A few structure types are used to +:c:type:`double` and :c:type:`char*`. A few structure types are used to describe static tables used to list the functions exported by a module or the data attributes of a new object type, and another is used to describe the value of a complex number. These will be discussed together with the functions that diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst index 7b179e22e290e..7bb0dad2b6b6d 100644 --- a/Doc/c-api/marshal.rst +++ b/Doc/c-api/marshal.rst @@ -43,7 +43,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: long PyMarshal_ReadLongFromFile(FILE *file) - Return a C :c:type:`long` from the data stream in a :c:type:`FILE\*` opened + Return a C :c:type:`long` from the data stream in a :c:type:`FILE*` opened for reading. Only a 32-bit value can be read in using this function, regardless of the native size of :c:type:`long`. @@ -53,7 +53,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: int PyMarshal_ReadShortFromFile(FILE *file) - Return a C :c:type:`short` from the data stream in a :c:type:`FILE\*` opened + Return a C :c:type:`short` from the data stream in a :c:type:`FILE*` opened for reading. Only a 16-bit value can be read in using this function, regardless of the native size of :c:type:`short`. @@ -63,7 +63,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: PyObject* PyMarshal_ReadObjectFromFile(FILE *file) - Return a Python object from the data stream in a :c:type:`FILE\*` opened for + Return a Python object from the data stream in a :c:type:`FILE*` opened for reading. On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError` @@ -72,7 +72,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file) - Return a Python object from the data stream in a :c:type:`FILE\*` opened for + Return a Python object from the data stream in a :c:type:`FILE*` opened for reading. Unlike :c:func:`PyMarshal_ReadObjectFromFile`, this function assumes that no further objects will be read from the file, allowing it to aggressively load file data into memory so that the de-serialization can diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 8a8542f0479ec..87425bcf1e71f 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -109,7 +109,7 @@ zero bytes. .. c:function:: void* PyMem_RawMalloc(size_t n) - Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as @@ -120,7 +120,7 @@ zero bytes. .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) Allocates *nelem* elements each whose size in bytes is *elsize* and returns - a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the + a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. The memory is initialized to zeros. Requesting zero elements or elements of size zero bytes returns a distinct @@ -180,7 +180,7 @@ The :ref:`default memory allocator ` uses the .. c:function:: void* PyMem_Malloc(size_t n) - Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as @@ -191,7 +191,7 @@ The :ref:`default memory allocator ` uses the .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) Allocates *nelem* elements each whose size in bytes is *elsize* and returns - a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the + a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. The memory is initialized to zeros. Requesting zero elements or elements of size zero bytes returns a distinct @@ -233,14 +233,14 @@ The following type-oriented macros are provided for convenience. Note that .. c:function:: TYPE* PyMem_New(TYPE, size_t n) Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of - memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have + memory. Returns a pointer cast to :c:type:`TYPE*`. The memory will not have been initialized in any way. .. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * - sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, + sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE*`. On return, *p* will be a pointer to the new memory area, or ``NULL`` in the event of failure. @@ -282,7 +282,7 @@ The :ref:`default object allocator ` uses the .. c:function:: void* PyObject_Malloc(size_t n) - Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as @@ -293,7 +293,7 @@ The :ref:`default object allocator ` uses the .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize) Allocates *nelem* elements each whose size in bytes is *elsize* and returns - a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the + a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. The memory is initialized to zeros. Requesting zero elements or elements of size zero bytes returns a distinct @@ -388,7 +388,7 @@ Customize Memory Allocators Enum used to identify an allocator domain. Domains: - .. c:var:: PYMEM_DOMAIN_RAW + .. c:macro:: PYMEM_DOMAIN_RAW Functions: @@ -397,7 +397,7 @@ Customize Memory Allocators * :c:func:`PyMem_RawCalloc` * :c:func:`PyMem_RawFree` - .. c:var:: PYMEM_DOMAIN_MEM + .. c:macro:: PYMEM_DOMAIN_MEM Functions: @@ -406,7 +406,7 @@ Customize Memory Allocators * :c:func:`PyMem_Calloc` * :c:func:`PyMem_Free` - .. c:var:: PYMEM_DOMAIN_OBJ + .. c:macro:: PYMEM_DOMAIN_OBJ Functions: @@ -519,11 +519,11 @@ Customize pymalloc Arena Allocator | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena | +--------------------------------------------------+---------------------------------------+ -.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) +.. c:function:: void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) Get the arena allocator. -.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) +.. c:function:: void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) Set the arena allocator. diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 8a415dfa30a35..6e9474bfa40eb 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -325,7 +325,7 @@ The *m_slots* array must be terminated by a slot with id 0. The available slot types are: -.. c:var:: Py_mod_create +.. c:macro:: Py_mod_create Specifies a function that is called to create the module object itself. The *value* pointer of this slot must point to a function of the signature: @@ -357,7 +357,7 @@ The available slot types are: ``PyModuleDef`` has non-``NULL`` ``m_traverse``, ``m_clear``, ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``. -.. c:var:: Py_mod_exec +.. c:macro:: Py_mod_exec Specifies a function that is called to *execute* the module. This is equivalent to executing the code of a Python module: typically, diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index b9c137ea352e7..a387b4a2df134 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -291,7 +291,7 @@ Object Protocol is equivalent to the Python expression ``type(o)``. This function increments the reference count of the return value. There's really no reason to use this function instead of the common expression ``o->ob_type``, which returns a - pointer of type :c:type:`PyTypeObject\*`, except when the incremented reference + pointer of type :c:type:`PyTypeObject*`, except when the incremented reference count is needed. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index b2392fa5e19c5..a9e1c6fbcc3f9 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -145,7 +145,7 @@ Implementing functions and methods .. c:type:: PyCFunction Type of the functions used to implement most Python callables in C. - Functions of this type take two :c:type:`PyObject\*` parameters and return + Functions of this type take two :c:type:`PyObject*` parameters and return one such value. If the return value is ``NULL``, an exception shall have been set. If not ``NULL``, the return value is interpreted as the return value of the function as exposed in Python. The function must return a new @@ -224,10 +224,10 @@ Implementing functions and methods +------------------+---------------+-------------------------------+ The :attr:`ml_meth` is a C function pointer. The functions may be of different -types, but they always return :c:type:`PyObject\*`. If the function is not of +types, but they always return :c:type:`PyObject*`. If the function is not of the :c:type:`PyCFunction`, the compiler will require a cast in the method table. Even though :c:type:`PyCFunction` defines the first parameter as -:c:type:`PyObject\*`, it is common that the method implementation uses the +:c:type:`PyObject*`, it is common that the method implementation uses the specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. @@ -239,7 +239,7 @@ There are these calling conventions: .. data:: METH_VARARGS This is the typical calling convention, where the methods have the type - :c:type:`PyCFunction`. The function expects two :c:type:`PyObject\*` values. + :c:type:`PyCFunction`. The function expects two :c:type:`PyObject*` values. The first one is the *self* object for methods; for module functions, it is the module object. The second parameter (often called *args*) is a tuple object representing all arguments. This parameter is typically processed @@ -260,7 +260,7 @@ There are these calling conventions: Fast calling convention supporting only positional arguments. The methods have the type :c:type:`_PyCFunctionFast`. The first parameter is *self*, the second parameter is a C array - of :c:type:`PyObject\*` values indicating the arguments and the third + of :c:type:`PyObject*` values indicating the arguments and the third parameter is the number of arguments (the length of the array). This is not part of the :ref:`limited API `. @@ -274,7 +274,7 @@ There are these calling conventions: with methods of type :c:type:`_PyCFunctionFastWithKeywords`. Keyword arguments are passed the same way as in the :ref:`vectorcall protocol `: - there is an additional fourth :c:type:`PyObject\*` parameter + there is an additional fourth :c:type:`PyObject*` parameter which is a tuple representing the names of the keyword arguments (which are guaranteed to be strings) or possibly ``NULL`` if there are no keywords. The values of the keyword @@ -312,7 +312,7 @@ There are these calling conventions: Methods with a single object argument can be listed with the :const:`METH_O` flag, instead of invoking :c:func:`PyArg_ParseTuple` with a ``"O"`` argument. They have the type :c:type:`PyCFunction`, with the *self* parameter, and a - :c:type:`PyObject\*` parameter representing the single argument. + :c:type:`PyObject*` parameter representing the single argument. These two constants are not used to indicate the calling convention but the @@ -463,7 +463,7 @@ Accessing attributes of extension types | | | getter and setter | +-------------+------------------+-----------------------------------+ - The ``get`` function takes one :c:type:`PyObject\*` parameter (the + The ``get`` function takes one :c:type:`PyObject*` parameter (the instance) and a function pointer (the associated ``closure``):: typedef PyObject *(*getter)(PyObject *, void *); @@ -471,7 +471,7 @@ Accessing attributes of extension types It should return a new reference on success or ``NULL`` with a set exception on failure. - ``set`` functions take two :c:type:`PyObject\*` parameters (the instance and + ``set`` functions take two :c:type:`PyObject*` parameters (the instance and the value to be set) and a function pointer (the associated ``closure``):: typedef int (*setter)(PyObject *, PyObject *, void *); diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst index c14cb2d38fd54..bf751e44acde0 100644 --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -161,7 +161,7 @@ type. .. c:type:: PyStructSequence_Field Describes a field of a struct sequence. As a struct sequence is modeled as a - tuple, all fields are typed as :c:type:`PyObject\*`. The index in the + tuple, all fields are typed as :c:type:`PyObject*`. The index in the :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which field of the struct sequence is described. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 385c7f94c672f..ddcb8ae3d0950 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -1348,7 +1348,7 @@ and :c:type:`PyType_Type` effectively act as defaults.) The following macro is defined to ease writing rich comparison functions: - .. c:function:: PyObject \*Py_RETURN_RICHCOMPARE(VAL_A, VAL_B, int op) + .. c:macro:: Py_RETURN_RICHCOMPARE(VAL_A, VAL_B, op) Return ``Py_True`` or ``Py_False`` from the function, depending on the result of a comparison. @@ -1386,7 +1386,7 @@ and :c:type:`PyType_Type` effectively act as defaults.) than zero and contains the offset in the instance structure of the weak reference list head (ignoring the GC header, if present); this offset is used by :c:func:`PyObject_ClearWeakRefs` and the :c:func:`PyWeakref_\*` functions. The - instance structure needs to include a field of type :c:type:`PyObject\*` which is + instance structure needs to include a field of type :c:type:`PyObject*` which is initialized to ``NULL``. Do not confuse this field with :c:member:`~PyTypeObject.tp_weaklist`; that is the list head for diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 577cdf2511466..5518214a793e0 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -199,7 +199,7 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 -.. c:function:: PyUnicode_MAX_CHAR_VALUE(PyObject *o) +.. c:macro:: PyUnicode_MAX_CHAR_VALUE(o) Return the maximum code point that is suitable for creating another string based on *o*, which must be in the "canonical" representation. This is diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index d6aecc394c9e7..b908cb8354f73 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -16,11 +16,11 @@ parameter. The available start symbols are :const:`Py_eval_input`, :const:`Py_file_input`, and :const:`Py_single_input`. These are described following the functions which accept them as parameters. -Note also that several of these functions take :c:type:`FILE\*` parameters. One +Note also that several of these functions take :c:type:`FILE*` parameters. One particular issue which needs to be handled carefully is that the :c:type:`FILE` structure for different C libraries can be different and incompatible. Under Windows (at least), it is possible for dynamically linked extensions to actually -use different libraries, so care should be taken that :c:type:`FILE\*` parameters +use different libraries, so care should be taken that :c:type:`FILE*` parameters are only passed to these functions if it is certain that they were created by the same library that the Python runtime is using. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index f1a033c6dae61..361e6db07c3cd 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -2309,9 +2309,9 @@ Encoders: :c:func:`PyUnicode_AsUTF8String` * :c:func:`PyUnicode_EncodeUTF32` * :c:func:`PyUnicode_EncodeUTF16` -* :c:func:`PyUnicode_EncodeUnicodeEscape:` use +* :c:func:`PyUnicode_EncodeUnicodeEscape` use :c:func:`PyUnicode_AsUnicodeEscapeString` -* :c:func:`PyUnicode_EncodeRawUnicodeEscape:` use +* :c:func:`PyUnicode_EncodeRawUnicodeEscape` use :c:func:`PyUnicode_AsRawUnicodeEscapeString` * :c:func:`PyUnicode_EncodeLatin1`: use :c:func:`PyUnicode_AsLatin1String` * :c:func:`PyUnicode_EncodeASCII`: use :c:func:`PyUnicode_AsASCIIString` From webhook-mailer at python.org Thu Aug 13 16:34:04 2020 From: webhook-mailer at python.org (Facundo Batista) Date: Thu, 13 Aug 2020 20:34:04 -0000 Subject: [Python-checkins] Fixed comment about pathlib.link_to: it was added in 3.8, not changed. (#21851) Message-ID: https://github.com/python/cpython/commit/a3eae43aeedb6e6a31adeab3c0c90961d05ab113 commit: a3eae43aeedb6e6a31adeab3c0c90961d05ab113 branch: master author: Facundo Batista committer: GitHub date: 2020-08-13T17:33:56-03:00 summary: Fixed comment about pathlib.link_to: it was added in 3.8, not changed. (#21851) files: M Doc/library/pathlib.rst diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index f705d15b8d4f1..04810f5204397 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1134,7 +1134,7 @@ call fails (for example because the path doesn't exist). Create a hard link pointing to a path named *target*. - .. versionchanged:: 3.8 + .. versionadded:: 3.8 .. method:: Path.write_bytes(data) From webhook-mailer at python.org Thu Aug 13 16:43:57 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Thu, 13 Aug 2020 20:43:57 -0000 Subject: [Python-checkins] Fixed comment about pathlib.link_to: it was added in 3.8, not changed. (GH-21851) (#21866) Message-ID: https://github.com/python/cpython/commit/e05f20b371ec8be939c0f79ef324dbc65bd56ecd commit: e05f20b371ec8be939c0f79ef324dbc65bd56ecd branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-13T17:43:47-03:00 summary: Fixed comment about pathlib.link_to: it was added in 3.8, not changed. (GH-21851) (#21866) (cherry picked from commit a3eae43aeedb6e6a31adeab3c0c90961d05ab113) Co-authored-by: Facundo Batista files: M Doc/library/pathlib.rst diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 6df352f0e3913..ebaedd835425c 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1078,7 +1078,7 @@ call fails (for example because the path doesn't exist). Create a hard link pointing to a path named *target*. - .. versionchanged:: 3.8 + .. versionadded:: 3.8 .. method:: Path.write_bytes(data) From webhook-mailer at python.org Thu Aug 13 19:36:02 2020 From: webhook-mailer at python.org (Steve Dower) Date: Thu, 13 Aug 2020 23:36:02 -0000 Subject: [Python-checkins] bpo-41526: Fixed layout of final page of the installer (GH-21871) Message-ID: https://github.com/python/cpython/commit/6444ca946984c638c67a72aac22fd6d3cc650c16 commit: 6444ca946984c638c67a72aac22fd6d3cc650c16 branch: master author: Steve Dower committer: GitHub date: 2020-08-14T00:35:52+01:00 summary: bpo-41526: Fixed layout of final page of the installer (GH-21871) files: A Misc/NEWS.d/next/Windows/2020-08-13-22-40-58.bpo-41526.-i2bwb.rst M Tools/msi/bundle/Default.thm M Tools/msi/bundle/Default.wxl diff --git a/Misc/NEWS.d/next/Windows/2020-08-13-22-40-58.bpo-41526.-i2bwb.rst b/Misc/NEWS.d/next/Windows/2020-08-13-22-40-58.bpo-41526.-i2bwb.rst new file mode 100644 index 0000000000000..756c8270599f2 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-08-13-22-40-58.bpo-41526.-i2bwb.rst @@ -0,0 +1,2 @@ +Fixed layout of final page of the installer by removing the special thanks +to Mark Hammond (with his permission). diff --git a/Tools/msi/bundle/Default.thm b/Tools/msi/bundle/Default.thm index 7c37d2cafb399..f5ba43d838fcf 100644 --- a/Tools/msi/bundle/Default.thm +++ b/Tools/msi/bundle/Default.thm @@ -116,9 +116,9 @@ #(loc.SuccessHeader) - + - + #(loc.SuccessRestartText) diff --git a/Tools/msi/bundle/Default.wxl b/Tools/msi/bundle/Default.wxl index f0088d4e13e6b..791ce6eab7474 100644 --- a/Tools/msi/bundle/Default.wxl +++ b/Tools/msi/bundle/Default.wxl @@ -107,9 +107,7 @@ Select Customize to review current options. &Restart New to Python? Start with the <a href="https://docs.python.org/[ShortVersion]/tutorial/index.html">online tutorial</a> and <a href="https://docs.python.org/[ShortVersion]/index.html">documentation</a>. At your terminal, type "py" to launch Python, or search for Python in your Start menu. -See <a href="https://docs.python.org/[ShortVersion]/whatsnew/[ShortVersion].html">what's new</a> in this release, or find more info about <a href="https://docs.python.org/[ShortVersion]/using/windows.html">using Python on Windows</a>. - -Special thanks to Mark Hammond, without whose years of freely shared Windows expertise, Python for Windows would still be Python for DOS. +See <a href="https://docs.python.org/[ShortVersion]/whatsnew/[ShortVersion].html">what's new</a> in this release, or find more info about <a href="https://docs.python.org/[ShortVersion]/using/windows.html">using Python on Windows</a>. Thank you for using [WixBundleName]. Thank you for using [WixBundleName]. From webhook-mailer at python.org Thu Aug 13 21:33:19 2020 From: webhook-mailer at python.org (Rishav Kundu) Date: Fri, 14 Aug 2020 01:33:19 -0000 Subject: [Python-checkins] bpo-41410: Fix outdated info in mkstemp docs (GH-21701) Message-ID: https://github.com/python/cpython/commit/e55de68be3e5b977a17d3c0ac9805b0feff8fedc commit: e55de68be3e5b977a17d3c0ac9805b0feff8fedc branch: master author: Rishav Kundu committer: GitHub date: 2020-08-13T18:33:14-07:00 summary: bpo-41410: Fix outdated info in mkstemp docs (GH-21701) Automerge-Triggered-By: @ericvsmith files: M Doc/library/tempfile.rst M Lib/tempfile.py diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index a59817c103921..3a2b88c0cb6a2 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -175,9 +175,8 @@ The module defines the following user-callable items: If you want to force a bytes return value with otherwise default behavior, pass ``suffix=b''``. - If *text* is specified, it indicates whether to open the file in binary - mode (the default) or text mode. On some platforms, this makes no - difference. + If *text* is specified and true, the file is opened in text mode. + Otherwise, (the default) the file is opened in binary mode. :func:`mkstemp` returns a tuple containing an OS-level handle to an open file (as would be returned by :func:`os.open`) and the absolute pathname diff --git a/Lib/tempfile.py b/Lib/tempfile.py index ba04be8f9058e..770f72c25295c 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -308,8 +308,7 @@ def mkstemp(suffix=None, prefix=None, dir=None, text=False): otherwise a default directory is used. If 'text' is specified and true, the file is opened in text - mode. Else (the default) the file is opened in binary mode. On - some operating systems, this makes no difference. + mode. Else (the default) the file is opened in binary mode. If any of 'suffix', 'prefix' and 'dir' are not None, they must be the same type. If they are bytes, the returned name will be bytes; str From webhook-mailer at python.org Thu Aug 13 21:51:29 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 14 Aug 2020 01:51:29 -0000 Subject: [Python-checkins] bpo-41410: Fix outdated info in mkstemp docs (GH-21701) Message-ID: https://github.com/python/cpython/commit/2a9f709ba23c8f6aa2bed821aacc4e7baecde383 commit: 2a9f709ba23c8f6aa2bed821aacc4e7baecde383 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-13T18:51:21-07:00 summary: bpo-41410: Fix outdated info in mkstemp docs (GH-21701) Automerge-Triggered-By: @ericvsmith (cherry picked from commit e55de68be3e5b977a17d3c0ac9805b0feff8fedc) Co-authored-by: Rishav Kundu files: M Doc/library/tempfile.rst M Lib/tempfile.py diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index a59817c103921..3a2b88c0cb6a2 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -175,9 +175,8 @@ The module defines the following user-callable items: If you want to force a bytes return value with otherwise default behavior, pass ``suffix=b''``. - If *text* is specified, it indicates whether to open the file in binary - mode (the default) or text mode. On some platforms, this makes no - difference. + If *text* is specified and true, the file is opened in text mode. + Otherwise, (the default) the file is opened in binary mode. :func:`mkstemp` returns a tuple containing an OS-level handle to an open file (as would be returned by :func:`os.open`) and the absolute pathname diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 5b990e067f23a..8f9cb6c3ca350 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -307,8 +307,7 @@ def mkstemp(suffix=None, prefix=None, dir=None, text=False): otherwise a default directory is used. If 'text' is specified and true, the file is opened in text - mode. Else (the default) the file is opened in binary mode. On - some operating systems, this makes no difference. + mode. Else (the default) the file is opened in binary mode. If any of 'suffix', 'prefix' and 'dir' are not None, they must be the same type. If they are bytes, the returned name will be bytes; str From webhook-mailer at python.org Thu Aug 13 22:38:40 2020 From: webhook-mailer at python.org (Paul Ganssle) Date: Fri, 14 Aug 2020 02:38:40 -0000 Subject: [Python-checkins] bpo-41025: Fix subclassing for zoneinfo.ZoneInfo (GH-20965) Message-ID: https://github.com/python/cpython/commit/87d8287865e5c9f137f6b5cf8c34c2c509eb5e9d commit: 87d8287865e5c9f137f6b5cf8c34c2c509eb5e9d branch: master author: Paul Ganssle committer: GitHub date: 2020-08-13T22:38:30-04:00 summary: bpo-41025: Fix subclassing for zoneinfo.ZoneInfo (GH-20965) Prior to this change, attempting to subclass the C implementation of zoneinfo.ZoneInfo gave the following error: TypeError: unbound method ZoneInfo.__init_subclass__() needs an argument https://bugs.python.org/issue41025 files: A Misc/NEWS.d/next/Library/2020-06-18-10-34-59.bpo-41025.elf_nz.rst M Lib/test/test_zoneinfo/test_zoneinfo.py M Modules/_zoneinfo.c diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index d16e0d2c33106..a9375fd55857b 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -463,7 +463,7 @@ class CZoneInfoDatetimeSubclassTest(DatetimeSubclassMixin, CZoneInfoTest): pass -class ZoneInfoTestSubclass(ZoneInfoTest): +class ZoneInfoSubclassTest(ZoneInfoTest): @classmethod def setUpClass(cls): super().setUpClass() @@ -484,7 +484,7 @@ def test_subclass_own_cache(self): self.assertIsInstance(sub_obj, self.klass) -class CZoneInfoTestSubclass(ZoneInfoTest): +class CZoneInfoSubclassTest(ZoneInfoSubclassTest): module = c_zoneinfo diff --git a/Misc/NEWS.d/next/Library/2020-06-18-10-34-59.bpo-41025.elf_nz.rst b/Misc/NEWS.d/next/Library/2020-06-18-10-34-59.bpo-41025.elf_nz.rst new file mode 100644 index 0000000000000..21e184d0a4063 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-06-18-10-34-59.bpo-41025.elf_nz.rst @@ -0,0 +1,2 @@ +Fixed an issue preventing the C implementation of :class:`zoneinfo.ZoneInfo` +from being subclassed. diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index bee84cbf8f9f4..12b3969959bac 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -2557,7 +2557,7 @@ static PyMethodDef zoneinfo_methods[] = { {"_unpickle", (PyCFunction)zoneinfo__unpickle, METH_VARARGS | METH_CLASS, PyDoc_STR("Private method used in unpickling.")}, {"__init_subclass__", (PyCFunction)(void (*)(void))zoneinfo_init_subclass, - METH_VARARGS | METH_KEYWORDS, + METH_VARARGS | METH_KEYWORDS | METH_CLASS, PyDoc_STR("Function to initialize subclasses.")}, {NULL} /* Sentinel */ }; From webhook-mailer at python.org Fri Aug 14 05:44:13 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 14 Aug 2020 09:44:13 -0000 Subject: [Python-checkins] bpo-33786: Fix asynchronous generators to handle GeneratorExit in athrow() (GH-7467) (GH-21878) Message-ID: https://github.com/python/cpython/commit/cf79cbf4479e395bf7c4df2907f5a444639b4f6f commit: cf79cbf4479e395bf7c4df2907f5a444639b4f6f branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-14T05:44:00-04:00 summary: bpo-33786: Fix asynchronous generators to handle GeneratorExit in athrow() (GH-7467) (GH-21878) (cherry picked from commit 52698c7ad9eae9feb35839fde17a7d1da8036a9b) Co-authored-by: Yury Selivanov files: A Misc/NEWS.d/next/Core and Builtins/2018-06-06-23-24-40.bpo-33786.lBvT8z.rst M Lib/contextlib.py M Lib/test/test_asyncgen.py M Lib/test/test_contextlib_async.py M Objects/genobject.c diff --git a/Lib/contextlib.py b/Lib/contextlib.py index 2d745ea3e3c55..13a3e1a5536c6 100644 --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -186,7 +186,7 @@ async def __aexit__(self, typ, value, traceback): # in this implementation try: await self.gen.athrow(typ, value, traceback) - raise RuntimeError("generator didn't stop after throw()") + raise RuntimeError("generator didn't stop after athrow()") except StopAsyncIteration as exc: return exc is not value except RuntimeError as exc: diff --git a/Lib/test/test_asyncgen.py b/Lib/test/test_asyncgen.py index 5a292fb5d64e1..5da27417f2d7d 100644 --- a/Lib/test/test_asyncgen.py +++ b/Lib/test/test_asyncgen.py @@ -108,6 +108,31 @@ def sync_iterate(g): res.append(str(type(ex))) return res + def async_iterate(g): + res = [] + while True: + an = g.__anext__() + try: + while True: + try: + an.__next__() + except StopIteration as ex: + if ex.args: + res.append(ex.args[0]) + break + else: + res.append('EMPTY StopIteration') + break + except StopAsyncIteration: + raise + except Exception as ex: + res.append(str(type(ex))) + break + except StopAsyncIteration: + res.append('STOP') + break + return res + def async_iterate(g): res = [] while True: @@ -297,6 +322,37 @@ async def gen(): "non-None value .* async generator"): gen().__anext__().send(100) + def test_async_gen_exception_11(self): + def sync_gen(): + yield 10 + yield 20 + + def sync_gen_wrapper(): + yield 1 + sg = sync_gen() + sg.send(None) + try: + sg.throw(GeneratorExit()) + except GeneratorExit: + yield 2 + yield 3 + + async def async_gen(): + yield 10 + yield 20 + + async def async_gen_wrapper(): + yield 1 + asg = async_gen() + await asg.asend(None) + try: + await asg.athrow(GeneratorExit()) + except GeneratorExit: + yield 2 + yield 3 + + self.compare_generators(sync_gen_wrapper(), async_gen_wrapper()) + def test_async_gen_api_01(self): async def gen(): yield 123 diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py index cc38dcf8c45c8..9db40652f40f5 100644 --- a/Lib/test/test_contextlib_async.py +++ b/Lib/test/test_contextlib_async.py @@ -36,6 +36,28 @@ async def __aexit__(self, *args): async with manager as context: self.assertIs(manager, context) + @_async_test + async def test_async_gen_propagates_generator_exit(self): + # A regression test for https://bugs.python.org/issue33786. + + @asynccontextmanager + async def ctx(): + yield + + async def gen(): + async with ctx(): + yield 11 + + ret = [] + exc = ValueError(22) + with self.assertRaises(ValueError): + async with ctx(): + async for val in gen(): + ret.append(val) + raise exc + + self.assertEqual(ret, [11]) + def test_exit_is_abstract(self): class MissingAexit(AbstractAsyncContextManager): pass diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-06-06-23-24-40.bpo-33786.lBvT8z.rst b/Misc/NEWS.d/next/Core and Builtins/2018-06-06-23-24-40.bpo-33786.lBvT8z.rst new file mode 100644 index 0000000000000..57deefe339b5c --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2018-06-06-23-24-40.bpo-33786.lBvT8z.rst @@ -0,0 +1 @@ +Fix asynchronous generators to handle GeneratorExit in athrow() correctly diff --git a/Objects/genobject.c b/Objects/genobject.c index dd7d44bd427df..b11690cde3e11 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1893,21 +1893,20 @@ async_gen_athrow_send(PyAsyncGenAThrow *o, PyObject *arg) return NULL; check_error: - if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration)) { + if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration) || + PyErr_ExceptionMatches(PyExc_GeneratorExit)) + { o->agt_state = AWAITABLE_STATE_CLOSED; if (o->agt_args == NULL) { /* when aclose() is called we don't want to propagate - StopAsyncIteration; just raise StopIteration, signalling - that 'aclose()' is done. */ + StopAsyncIteration or GeneratorExit; just raise + StopIteration, signalling that this 'aclose()' await + is done. + */ PyErr_Clear(); PyErr_SetNone(PyExc_StopIteration); } } - else if (PyErr_ExceptionMatches(PyExc_GeneratorExit)) { - o->agt_state = AWAITABLE_STATE_CLOSED; - PyErr_Clear(); /* ignore these errors */ - PyErr_SetNone(PyExc_StopIteration); - } return NULL; } From webhook-mailer at python.org Fri Aug 14 06:20:37 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Fri, 14 Aug 2020 10:20:37 -0000 Subject: [Python-checkins] bpo-40204: Fix reference to terms in the doc (GH-21865) Message-ID: https://github.com/python/cpython/commit/bb0b08540cc93e56f3f1bde1b39ce086d9e35fe1 commit: bb0b08540cc93e56f3f1bde1b39ce086d9e35fe1 branch: master author: Victor Stinner committer: GitHub date: 2020-08-14T12:20:05+02:00 summary: bpo-40204: Fix reference to terms in the doc (GH-21865) Sphinx 3 requires to refer to terms with the exact case. For example, fix the Sphinx 3 warning: Doc/library/pkgutil.rst:71: WARNING: term Loader not found in case sensitive match.made a reference to loader instead. files: M Doc/extending/newtypes_tutorial.rst M Doc/glossary.rst M Doc/library/collections.abc.rst M Doc/library/concurrent.futures.rst M Doc/library/importlib.rst M Doc/library/multiprocessing.rst M Doc/library/pkgutil.rst M Doc/library/threading.rst M Doc/reference/datamodel.rst M Doc/tutorial/classes.rst M Doc/whatsnew/3.2.rst M Doc/whatsnew/3.5.rst M Doc/whatsnew/3.6.rst M Doc/whatsnew/3.7.rst diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 0eb6ffd026f49..4da77e797d222 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -416,7 +416,7 @@ But this would be risky. Our type doesn't restrict the type of the ``first`` member, so it could be any kind of object. It could have a destructor that causes code to be executed that tries to access the ``first`` member; or that destructor could release the -:term:`Global interpreter Lock` and let arbitrary code run in other +:term:`Global interpreter Lock ` and let arbitrary code run in other threads that accesses and modifies our object. To be paranoid and protect ourselves against this possibility, we almost diff --git a/Doc/glossary.rst b/Doc/glossary.rst index e997d366777b3..7be755e411310 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -587,7 +587,7 @@ Glossary and :class:`tuple`) and some non-sequence types like :class:`dict`, :term:`file objects `, and objects of any classes you define with an :meth:`__iter__` method or with a :meth:`__getitem__` method - that implements :term:`Sequence` semantics. + that implements :term:`Sequence ` semantics. Iterables can be used in a :keyword:`for` loop and in many other places where a sequence is diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 2a3fb142f7297..dc7ae30b6d2fa 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -185,7 +185,7 @@ ABC Inherits from Abstract Methods Mixin expressions. Custom implementations must provide the :meth:`__await__` method. - :term:`Coroutine` objects and instances of the + :term:`Coroutine ` objects and instances of the :class:`~collections.abc.Coroutine` ABC are all instances of this ABC. .. note:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index b21d5594c84fa..675a9ffdd0711 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -221,7 +221,8 @@ ProcessPoolExecutor The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that uses a pool of processes to execute calls asynchronously. :class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which -allows it to side-step the :term:`Global Interpreter Lock` but also means that +allows it to side-step the :term:`Global Interpreter Lock +` but also means that only picklable objects can be executed and returned. The ``__main__`` module must be importable by worker subprocesses. This means diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index f7286d2ade8bd..5fb0a4a120b98 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -1073,7 +1073,7 @@ find and load modules. .. class:: WindowsRegistryFinder - :term:`Finder` for modules declared in the Windows registry. This class + :term:`Finder ` for modules declared in the Windows registry. This class implements the :class:`importlib.abc.MetaPathFinder` ABC. Only class methods are defined by this class to alleviate the need for @@ -1088,7 +1088,7 @@ find and load modules. .. class:: PathFinder - A :term:`Finder` for :data:`sys.path` and package ``__path__`` attributes. + A :term:`Finder ` for :data:`sys.path` and package ``__path__`` attributes. This class implements the :class:`importlib.abc.MetaPathFinder` ABC. Only class methods are defined by this class to alleviate the need for diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 69d6523650386..28510acd52bd4 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -14,7 +14,8 @@ Introduction :mod:`multiprocessing` is a package that supports spawning processes using an API similar to the :mod:`threading` module. The :mod:`multiprocessing` package offers both local and remote concurrency, effectively side-stepping the -:term:`Global Interpreter Lock` by using subprocesses instead of threads. Due +:term:`Global Interpreter Lock ` by using +subprocesses instead of threads. Due to this, the :mod:`multiprocessing` module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows. diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 2066cbb9fc57c..3b17b9a621987 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -68,7 +68,7 @@ support. .. class:: ImpLoader(fullname, file, filename, etc) - :term:`Loader` that wraps Python's "classic" import algorithm. + :term:`Loader ` that wraps Python's "classic" import algorithm. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 458e39bf721c6..7fcf93d74610e 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -394,7 +394,8 @@ since it is impossible to detect the termination of alien threads. .. impl-detail:: - In CPython, due to the :term:`Global Interpreter Lock`, only one thread + In CPython, due to the :term:`Global Interpreter Lock + `, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index c5a7f046992dd..a817408c3b1ef 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2601,7 +2601,7 @@ Awaitable Objects ----------------- An :term:`awaitable` object generally implements an :meth:`__await__` method. -:term:`Coroutine` objects returned from :keyword:`async def` functions +:term:`Coroutine objects ` returned from :keyword:`async def` functions are awaitable. .. note:: @@ -2626,7 +2626,7 @@ are awaitable. Coroutine Objects ----------------- -:term:`Coroutine` objects are :term:`awaitable` objects. +:term:`Coroutine objects ` are :term:`awaitable` objects. A coroutine's execution can be controlled by calling :meth:`__await__` and iterating over the result. When the coroutine has finished executing and returns, the iterator raises :exc:`StopIteration`, and the exception's diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index 685552f99f440..0d780e3ba8964 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -849,7 +849,7 @@ defines :meth:`__next__`, then :meth:`__iter__` can just return ``self``:: Generators ========== -:term:`Generator`\s are a simple and powerful tool for creating iterators. They +:term:`Generators ` are a simple and powerful tool for creating iterators. They are written like regular functions but use the :keyword:`yield` statement whenever they want to return data. Each time :func:`next` is called on it, the generator resumes where it left off (it remembers all the data values and which diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst index ca3eda05c515a..06bee9966c0be 100644 --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -2311,7 +2311,7 @@ Multi-threading =============== * The mechanism for serializing execution of concurrently running Python threads - (generally known as the :term:`GIL` or :term:`Global Interpreter Lock`) has + (generally known as the :term:`GIL` or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a "check interval" to allow thread diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst index b4540ac1dd902..1defee4090f28 100644 --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -412,7 +412,7 @@ uses were to provide type hints to function parameters and return values. It became evident that it would be beneficial for Python users, if the standard library included the base definitions and tools for type annotations. -:pep:`484` introduces a :term:`provisional module ` to +:pep:`484` introduces a :term:`provisional module ` to provide these standard definitions and tools, along with some conventions for situations where annotations are not available. @@ -726,7 +726,7 @@ New Modules typing ------ -The new :mod:`typing` :term:`provisional ` module +The new :mod:`typing` :term:`provisional ` module provides standard definitions and tools for function type annotations. See :ref:`Type Hints ` for more information. @@ -772,7 +772,7 @@ Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) asyncio ------- -Since the :mod:`asyncio` module is :term:`provisional `, +Since the :mod:`asyncio` module is :term:`provisional `, all changes introduced in Python 3.5 have also been backported to Python 3.4.x. Notable changes in the :mod:`asyncio` module since Python 3.4.0: @@ -1867,7 +1867,7 @@ A new :func:`~sys.set_coroutine_wrapper` function allows setting a global hook that will be called whenever a :term:`coroutine object ` is created by an :keyword:`async def` function. A corresponding :func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set -wrapper. Both functions are :term:`provisional `, +wrapper. Both functions are :term:`provisional `, and are intended for debugging purposes only. (Contributed by Yury Selivanov in :issue:`24017`.) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 04c1f7e71db32..85a6657fdfbda 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -1597,7 +1597,7 @@ to filter block traces by their address space (domain). typing ------ -Since the :mod:`typing` module is :term:`provisional `, +Since the :mod:`typing` module is :term:`provisional `, all changes introduced in Python 3.6 have also been backported to Python 3.5.x. diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 279bbc697b5c6..25b1e1e33e325 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -636,7 +636,7 @@ The :mod:`asyncio` module has received many new features, usability and :ref:`performance improvements `. Notable changes include: -* The new :term:`provisional ` :func:`asyncio.run` function can +* The new :term:`provisional ` :func:`asyncio.run` function can be used to run a coroutine from synchronous code by automatically creating and destroying the event loop. (Contributed by Yury Selivanov in :issue:`32314`.) From webhook-mailer at python.org Fri Aug 14 19:01:45 2020 From: webhook-mailer at python.org (Irit Katriel) Date: Fri, 14 Aug 2020 23:01:45 -0000 Subject: [Python-checkins] Fix typo in typing doc (GH-21879) Message-ID: https://github.com/python/cpython/commit/fa5d7251987c70a9c5d58b59a0b36ac9287eaafa commit: fa5d7251987c70a9c5d58b59a0b36ac9287eaafa branch: master author: Irit Katriel committer: GitHub date: 2020-08-14T16:01:36-07:00 summary: Fix typo in typing doc (GH-21879) Automerge-Triggered-By: @gvanrossum files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 44b537f1669e1..8208680669de6 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -325,7 +325,7 @@ every type as being compatible with :data:`Any` and :data:`Any` as being compatible with every type. This means that it is possible to perform any operation or method call on a -value of type on :data:`Any` and assign it to any variable:: +value of type :data:`Any` and assign it to any variable:: from typing import Any From webhook-mailer at python.org Fri Aug 14 19:10:08 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 14 Aug 2020 23:10:08 -0000 Subject: [Python-checkins] Fix typo in typing doc (GH-21879) Message-ID: https://github.com/python/cpython/commit/0a5b30d98913e84f80ecea2b861e96d8f67c89e9 commit: 0a5b30d98913e84f80ecea2b861e96d8f67c89e9 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-14T16:09:58-07:00 summary: Fix typo in typing doc (GH-21879) Automerge-Triggered-By: @gvanrossum (cherry picked from commit fa5d7251987c70a9c5d58b59a0b36ac9287eaafa) Co-authored-by: Irit Katriel files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 1467276d0141d..405562ba2a8f8 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -324,7 +324,7 @@ every type as being compatible with :data:`Any` and :data:`Any` as being compatible with every type. This means that it is possible to perform any operation or method call on a -value of type on :data:`Any` and assign it to any variable:: +value of type :data:`Any` and assign it to any variable:: from typing import Any From webhook-mailer at python.org Sat Aug 15 10:06:37 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Sat, 15 Aug 2020 14:06:37 -0000 Subject: [Python-checkins] bpo-40878: xlc cannot handle C99 extern inline. (GH-21887) Message-ID: https://github.com/python/cpython/commit/40e700ad042089120456cc2ee79b8ca69479416b commit: 40e700ad042089120456cc2ee79b8ca69479416b branch: master author: Stefan Krah committer: GitHub date: 2020-08-15T16:06:21+02:00 summary: bpo-40878: xlc cannot handle C99 extern inline. (GH-21887) This applies to the default "extc99" mode. Python does not compile with "stdc99". files: M Modules/_decimal/libmpdec/mpdecimal.c diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c index b3ec13a509bb4..f0e4d7f343a43 100644 --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -64,7 +64,7 @@ #if defined(_MSC_VER) #define ALWAYS_INLINE __forceinline -#elif defined(LEGACY_COMPILER) +#elif defined(__IBMC__) || defined(LEGACY_COMPILER) #define ALWAYS_INLINE #undef inline #define inline From webhook-mailer at python.org Sat Aug 15 13:01:40 2020 From: webhook-mailer at python.org (Dima Tisnek) Date: Sat, 15 Aug 2020 17:01:40 -0000 Subject: [Python-checkins] bpo-31122: ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation (GH-18772) Message-ID: https://github.com/python/cpython/commit/495bd035662fda29639f9d52bb6baebea31d72fa commit: 495bd035662fda29639f9d52bb6baebea31d72fa branch: master author: Dima Tisnek committer: GitHub date: 2020-08-15T10:01:19-07:00 summary: bpo-31122: ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation (GH-18772) [bpo-31122](): ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation Reproducer: http://tiny.cc/f4ztnz (tiny url because some bot keeps renaming b.p.o.-nnn as bpo links) files: A Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst M Modules/_ssl.c diff --git a/Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst b/Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst new file mode 100644 index 0000000000000..2e70f7aee65c8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst @@ -0,0 +1 @@ +ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation \ No newline at end of file diff --git a/Modules/_ssl.c b/Modules/_ssl.c index 55a95ddf774e6..cb8f04a900a06 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -805,10 +805,11 @@ PySSL_SetError(PySSLSocket *sslsock, int ret, const char *filename, int lineno) errno = err.c; return PyErr_SetFromErrno(PyExc_OSError); } - Py_INCREF(s); - s->errorhandler(); - Py_DECREF(s); - return NULL; + else { + p = PY_SSL_ERROR_EOF; + type = PySSLEOFErrorObject; + errstr = "EOF occurred in violation of protocol"; + } } else { /* possible? */ p = PY_SSL_ERROR_SYSCALL; type = PySSLSyscallErrorObject; From webhook-mailer at python.org Sat Aug 15 13:42:40 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 15 Aug 2020 17:42:40 -0000 Subject: [Python-checkins] bpo-31122: ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation (GH-18772) Message-ID: https://github.com/python/cpython/commit/243458115e2cb295fb5bbb61e6ac528c6b2cf5be commit: 243458115e2cb295fb5bbb61e6ac528c6b2cf5be branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-15T10:42:36-07:00 summary: bpo-31122: ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation (GH-18772) [bpo-31122](): ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation Reproducer: http://tiny.cc/f4ztnz (tiny url because some bot keeps renaming b.p.o.-nnn as bpo links) (cherry picked from commit 495bd035662fda29639f9d52bb6baebea31d72fa) Co-authored-by: Dima Tisnek files: A Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst M Modules/_ssl.c diff --git a/Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst b/Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst new file mode 100644 index 0000000000000..2e70f7aee65c8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-03-11-07-44-06.bpo-31122.zIQ80l.rst @@ -0,0 +1 @@ +ssl.wrap_socket() now raises ssl.SSLEOFError rather than OSError when peer closes connection during TLS negotiation \ No newline at end of file diff --git a/Modules/_ssl.c b/Modules/_ssl.c index 1944393e54890..e6dda298cbe25 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -807,10 +807,11 @@ PySSL_SetError(PySSLSocket *sslsock, int ret, const char *filename, int lineno) errno = err.c; return PyErr_SetFromErrno(PyExc_OSError); } - Py_INCREF(s); - s->errorhandler(); - Py_DECREF(s); - return NULL; + else { + p = PY_SSL_ERROR_EOF; + type = PySSLEOFErrorObject; + errstr = "EOF occurred in violation of protocol"; + } } else { /* possible? */ p = PY_SSL_ERROR_SYSCALL; type = PySSLSyscallErrorObject; From webhook-mailer at python.org Sat Aug 15 14:19:16 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Sat, 15 Aug 2020 18:19:16 -0000 Subject: [Python-checkins] bpo-41540: AIX: skip test that is flaky with a default ulimit. (#21890) Message-ID: https://github.com/python/cpython/commit/39dab24621122338d01c1219bb0acc46ba9c9956 commit: 39dab24621122338d01c1219bb0acc46ba9c9956 branch: master author: Stefan Krah committer: GitHub date: 2020-08-15T20:19:07+02:00 summary: bpo-41540: AIX: skip test that is flaky with a default ulimit. (#21890) - AIX has extreme over-allocation that is in no relation to the physical RAM and swap. files: M Lib/test/test_decimal.py diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 113b37ddaa9cd..dbd58e8a6519b 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -5661,6 +5661,9 @@ def __abs__(self): self.assertEqual(Decimal.from_float(cls(101.1)), Decimal.from_float(101.1)) + # Issue 41540: + @unittest.skipIf(sys.platform.startswith("aix"), + "AIX: default ulimit: test is flaky because of extreme over-allocation") def test_maxcontext_exact_arith(self): # Make sure that exact operations do not raise MemoryError due From webhook-mailer at python.org Sat Aug 15 22:38:25 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Sun, 16 Aug 2020 02:38:25 -0000 Subject: [Python-checkins] bpo-41513: Improve speed and accuracy of math.hypot() (GH-21803) Message-ID: https://github.com/python/cpython/commit/fff3c28052e6b0750d6218e00acacd2fded4991a commit: fff3c28052e6b0750d6218e00acacd2fded4991a branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-15T19:38:19-07:00 summary: bpo-41513: Improve speed and accuracy of math.hypot() (GH-21803) files: A Misc/NEWS.d/next/Library/2020-08-09-18-16-05.bpo-41513.e6K6EK.rst M Lib/test/test_math.py M Modules/mathmodule.c diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py index e06b1e6a5b9b7..4d62eb1b11993 100644 --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -795,7 +795,8 @@ def testHypot(self): # Verify scaling for extremely large values fourthmax = FLOAT_MAX / 4.0 for n in range(32): - self.assertEqual(hypot(*([fourthmax]*n)), fourthmax * math.sqrt(n)) + self.assertTrue(math.isclose(hypot(*([fourthmax]*n)), + fourthmax * math.sqrt(n))) # Verify scaling for extremely small values for exp in range(32): @@ -904,8 +905,8 @@ class T(tuple): for n in range(32): p = (fourthmax,) * n q = (0.0,) * n - self.assertEqual(dist(p, q), fourthmax * math.sqrt(n)) - self.assertEqual(dist(q, p), fourthmax * math.sqrt(n)) + self.assertTrue(math.isclose(dist(p, q), fourthmax * math.sqrt(n))) + self.assertTrue(math.isclose(dist(q, p), fourthmax * math.sqrt(n))) # Verify scaling for extremely small values for exp in range(32): diff --git a/Misc/NEWS.d/next/Library/2020-08-09-18-16-05.bpo-41513.e6K6EK.rst b/Misc/NEWS.d/next/Library/2020-08-09-18-16-05.bpo-41513.e6K6EK.rst new file mode 100644 index 0000000000000..cfb9f98c376a0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-09-18-16-05.bpo-41513.e6K6EK.rst @@ -0,0 +1,2 @@ +Minor algorithmic improvement to math.hypot() and math.dist() giving small +gains in speed and accuracy. diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 411c6eb1935fa..489802cc36745 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2406,6 +2406,13 @@ math_fmod_impl(PyObject *module, double x, double y) /* Given an *n* length *vec* of values and a value *max*, compute: + sqrt(sum((x * scale) ** 2 for x in vec)) / scale + + where scale is the first power of two + greater than max. + +or compute: + max * sqrt(sum((x / max) ** 2 for x in vec)) The value of the *max* variable must be non-negative and @@ -2425,19 +2432,25 @@ The *csum* variable tracks the cumulative sum and *frac* tracks the cumulative fractional errors at each step. Since this variant assumes that |csum| >= |x| at each step, we establish the precondition by starting the accumulation from 1.0 which -represents the largest possible value of (x/max)**2. +represents the largest possible value of (x*scale)**2 or (x/max)**2. After the loop is finished, the initial 1.0 is subtracted out for a net zero effect on the final sum. Since *csum* will be greater than 1.0, the subtraction of 1.0 will not cause fractional digits to be dropped from *csum*. +To get the full benefit from compensated summation, the +largest addend should be in the range: 0.5 <= x <= 1.0. +Accordingly, scaling or division by *max* should not be skipped +even if not otherwise needed to prevent overflow or loss of precision. + */ static inline double vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) { - double x, csum = 1.0, oldcsum, frac = 0.0; + double x, csum = 1.0, oldcsum, frac = 0.0, scale; + int max_e; Py_ssize_t i; if (Py_IS_INFINITY(max)) { @@ -2449,14 +2462,36 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) if (max == 0.0 || n <= 1) { return max; } + frexp(max, &max_e); + if (max_e >= -1023) { + scale = ldexp(1.0, -max_e); + assert(max * scale >= 0.5); + assert(max * scale < 1.0); + for (i=0 ; i < n ; i++) { + x = vec[i]; + assert(Py_IS_FINITE(x) && fabs(x) <= max); + x *= scale; + x = x*x; + assert(x <= 1.0); + assert(csum >= x); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + } + return sqrt(csum - 1.0 + frac) / scale; + } + /* When max_e < -1023, ldexp(1.0, -max_e) overflows. + So instead of multiplying by a scale, we just divide by *max*. + */ for (i=0 ; i < n ; i++) { x = vec[i]; assert(Py_IS_FINITE(x) && fabs(x) <= max); x /= max; x = x*x; + assert(x <= 1.0); + assert(csum >= x); oldcsum = csum; csum += x; - assert(csum >= x); frac += (oldcsum - csum) + x; } return max * sqrt(csum - 1.0 + frac); From webhook-mailer at python.org Sun Aug 16 11:10:22 2020 From: webhook-mailer at python.org (Irit Katriel) Date: Sun, 16 Aug 2020 15:10:22 -0000 Subject: [Python-checkins] bpo-41503: Fix race between setTarget and flush in logging.handlers.MemoryHandler (GH-21765) Message-ID: https://github.com/python/cpython/commit/2353d77fad7ed9d11d8a4d66b5dd1306cdb94125 commit: 2353d77fad7ed9d11d8a4d66b5dd1306cdb94125 branch: master author: Irit Katriel committer: GitHub date: 2020-08-16T16:10:13+01:00 summary: bpo-41503: Fix race between setTarget and flush in logging.handlers.MemoryHandler (GH-21765) files: A Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst M Lib/logging/handlers.py M Lib/test/test_logging.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index 4a120e9f1ec48..867ef4ebc7600 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -1324,7 +1324,11 @@ def setTarget(self, target): """ Set the target handler for this handler. """ - self.target = target + self.acquire() + try: + self.target = target + finally: + self.release() def flush(self): """ diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index eb5b926908a19..d8b3727eb1185 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1160,6 +1160,27 @@ def test_flush_on_close(self): # assert that no new lines have been added self.assert_log_lines(lines) # no change + def test_race_between_set_target_and_flush(self): + class MockRaceConditionHandler: + def __init__(self, mem_hdlr): + self.mem_hdlr = mem_hdlr + + def removeTarget(self): + self.mem_hdlr.setTarget(None) + + def handle(self, msg): + t = threading.Thread(target=self.removeTarget) + t.daemon = True + t.start() + + target = MockRaceConditionHandler(self.mem_hdlr) + self.mem_hdlr.setTarget(target) + + for _ in range(10): + time.sleep(0.005) + self.mem_logger.info("not flushed") + self.mem_logger.warning("flushed") + class ExceptionFormatter(logging.Formatter): """A special exception formatter.""" diff --git a/Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst b/Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst new file mode 100644 index 0000000000000..c34996d881937 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst @@ -0,0 +1 @@ +Fixed a race between setTarget and flush in logging.handlers.MemoryHandler. \ No newline at end of file From webhook-mailer at python.org Sun Aug 16 11:51:05 2020 From: webhook-mailer at python.org (Soumendra Ganguly) Date: Sun, 16 Aug 2020 15:51:05 -0000 Subject: [Python-checkins] Update the comment of termios.c (#21886) Message-ID: https://github.com/python/cpython/commit/97003466d481670b9234b343319c6e02f9c5d732 commit: 97003466d481670b9234b343319c6e02f9c5d732 branch: master author: Soumendra Ganguly <67527439+8vasu at users.noreply.github.com> committer: GitHub date: 2020-08-17T00:51:00+09:00 summary: Update the comment of termios.c (#21886) files: M Modules/termios.c diff --git a/Modules/termios.c b/Modules/termios.c index 75e5e523206f4..178ae4ee6e41d 100644 --- a/Modules/termios.c +++ b/Modules/termios.c @@ -1,4 +1,4 @@ -/* termiosmodule.c -- POSIX terminal I/O module implementation. */ +/* termios.c -- POSIX terminal I/O module implementation. */ #include "Python.h" From webhook-mailer at python.org Sun Aug 16 12:27:09 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 16 Aug 2020 16:27:09 -0000 Subject: [Python-checkins] [3.8] bpo-41503: Fix race between setTarget and flush in logging.handlers.MemoryHandler (GH-21765) (GH-21898) Message-ID: https://github.com/python/cpython/commit/08f0a2109297e8a64e8636d47dce737e5b7ccf2c commit: 08f0a2109297e8a64e8636d47dce737e5b7ccf2c branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-16T09:27:01-07:00 summary: [3.8] bpo-41503: Fix race between setTarget and flush in logging.handlers.MemoryHandler (GH-21765) (GH-21898) (cherry picked from commit 2353d77fad7ed9d11d8a4d66b5dd1306cdb94125) Co-authored-by: Irit Katriel Automerge-Triggered-By: @vsajip files: A Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst M Lib/logging/handlers.py M Lib/test/test_logging.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index 34ff7a056ef54..29802b61d1c1f 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -1294,7 +1294,11 @@ def setTarget(self, target): """ Set the target handler for this handler. """ - self.target = target + self.acquire() + try: + self.target = target + finally: + self.release() def flush(self): """ diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 09b273bf1fb2d..cea51b455eed2 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1173,6 +1173,27 @@ def test_flush_on_close(self): # assert that no new lines have been added self.assert_log_lines(lines) # no change + def test_race_between_set_target_and_flush(self): + class MockRaceConditionHandler: + def __init__(self, mem_hdlr): + self.mem_hdlr = mem_hdlr + + def removeTarget(self): + self.mem_hdlr.setTarget(None) + + def handle(self, msg): + t = threading.Thread(target=self.removeTarget) + t.daemon = True + t.start() + + target = MockRaceConditionHandler(self.mem_hdlr) + self.mem_hdlr.setTarget(target) + + for _ in range(10): + time.sleep(0.005) + self.mem_logger.info("not flushed") + self.mem_logger.warning("flushed") + class ExceptionFormatter(logging.Formatter): """A special exception formatter.""" diff --git a/Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst b/Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst new file mode 100644 index 0000000000000..c34996d881937 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-07-15-18-16.bpo-41503.IYftcu.rst @@ -0,0 +1 @@ +Fixed a race between setTarget and flush in logging.handlers.MemoryHandler. \ No newline at end of file From webhook-mailer at python.org Mon Aug 17 01:20:50 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 17 Aug 2020 05:20:50 -0000 Subject: [Python-checkins] bpo-41521: Rename blacklist parameter to not_exported (GH-21824) Message-ID: https://github.com/python/cpython/commit/fbf43f051e7bf479709e122efa4b6edd4b09d4df commit: fbf43f051e7bf479709e122efa4b6edd4b09d4df branch: master author: Victor Stinner committer: GitHub date: 2020-08-17T07:20:40+02:00 summary: bpo-41521: Rename blacklist parameter to not_exported (GH-21824) Rename "blacklist" parameter of test.support.check__all__() to "not_exported". files: A Misc/NEWS.d/next/Tests/2020-08-11-14-59-13.bpo-41521.w2UYK7.rst M Doc/library/test.rst M Lib/test/_test_multiprocessing.py M Lib/test/support/__init__.py M Lib/test/test_calendar.py M Lib/test/test_cgi.py M Lib/test/test_configparser.py M Lib/test/test_ftplib.py M Lib/test/test_gettext.py M Lib/test/test_logging.py M Lib/test/test_mailbox.py M Lib/test/test_optparse.py M Lib/test/test_pickletools.py M Lib/test/test_plistlib.py M Lib/test/test_smtpd.py M Lib/test/test_support.py M Lib/test/test_tarfile.py M Lib/test/test_threading.py M Lib/test/test_wave.py M Lib/test/test_xml_etree.py diff --git a/Doc/library/test.rst b/Doc/library/test.rst index cd05ef07b4a21..6495b4844449e 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -878,7 +878,7 @@ The :mod:`test.support` module defines the following functions: missing. -.. function:: check__all__(test_case, module, name_of_module=None, extra=(), blacklist=()) +.. function:: check__all__(test_case, module, name_of_module=None, extra=(), not_exported=()) Assert that the ``__all__`` variable of *module* contains all public names. @@ -895,8 +895,8 @@ The :mod:`test.support` module defines the following functions: detected as "public", like objects without a proper ``__module__`` attribute. If provided, it will be added to the automatically detected ones. - The *blacklist* argument can be a set of names that must not be treated as part of - the public API even though their names indicate otherwise. + The *not_exported* argument can be a set of names that must not be treated + as part of the public API even though their names indicate otherwise. Example use:: @@ -912,10 +912,10 @@ The :mod:`test.support` module defines the following functions: class OtherTestCase(unittest.TestCase): def test__all__(self): extra = {'BAR_CONST', 'FOO_CONST'} - blacklist = {'baz'} # Undocumented name. + not_exported = {'baz'} # Undocumented name. # bar imports part of its API from _bar. support.check__all__(self, bar, ('bar', '_bar'), - extra=extra, blacklist=blacklist) + extra=extra, not_exported=not_exported) .. versionadded:: 3.6 diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 58663c02002e9..6a0e1016aff8d 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -5581,9 +5581,11 @@ def test_namespace(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - # Just make sure names in blacklist are excluded + # Just make sure names in not_exported are excluded support.check__all__(self, multiprocessing, extra=multiprocessing.__all__, - blacklist=['SUBDEBUG', 'SUBWARNING']) + not_exported=['SUBDEBUG', 'SUBWARNING']) + + # # Mixins # diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index e9573d1335210..4ba749454c187 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1410,7 +1410,7 @@ def detect_api_mismatch(ref_api, other_api, *, ignore=()): def check__all__(test_case, module, name_of_module=None, extra=(), - blacklist=()): + not_exported=()): """Assert that the __all__ variable of 'module' contains all public names. The module's public names (its API) are detected automatically based on @@ -1427,7 +1427,7 @@ def check__all__(test_case, module, name_of_module=None, extra=(), '__module__' attribute. If provided, it will be added to the automatically detected ones. - The 'blacklist' argument can be a set of names that must not be treated + The 'not_exported' argument can be a set of names that must not be treated as part of the public API even though their names indicate otherwise. Usage: @@ -1443,10 +1443,10 @@ def test__all__(self): class OtherTestCase(unittest.TestCase): def test__all__(self): extra = {'BAR_CONST', 'FOO_CONST'} - blacklist = {'baz'} # Undocumented name. + not_exported = {'baz'} # Undocumented name. # bar imports part of its API from _bar. support.check__all__(self, bar, ('bar', '_bar'), - extra=extra, blacklist=blacklist) + extra=extra, not_exported=not_exported) """ @@ -1458,7 +1458,7 @@ def test__all__(self): expected = set(extra) for name in dir(module): - if name.startswith('_') or name in blacklist: + if name.startswith('_') or name in not_exported: continue obj = getattr(module, name) if (getattr(obj, '__module__', None) in name_of_module or diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py index 7c7ec1c931aa4..c641e8c418318 100644 --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -934,12 +934,12 @@ def test_html_output_year_css(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {'mdays', 'January', 'February', 'EPOCH', - 'MONDAY', 'TUESDAY', 'WEDNESDAY', 'THURSDAY', 'FRIDAY', - 'SATURDAY', 'SUNDAY', 'different_locale', 'c', - 'prweek', 'week', 'format', 'formatstring', 'main', - 'monthlen', 'prevmonth', 'nextmonth'} - support.check__all__(self, calendar, blacklist=blacklist) + not_exported = { + 'mdays', 'January', 'February', 'EPOCH', 'MONDAY', 'TUESDAY', + 'WEDNESDAY', 'THURSDAY', 'FRIDAY', 'SATURDAY', 'SUNDAY', + 'different_locale', 'c', 'prweek', 'week', 'format', + 'formatstring', 'main', 'monthlen', 'prevmonth', 'nextmonth'} + support.check__all__(self, calendar, not_exported=not_exported) class TestSubClassingCase(unittest.TestCase): diff --git a/Lib/test/test_cgi.py b/Lib/test/test_cgi.py index 101942de947fb..6b29759da44d0 100644 --- a/Lib/test/test_cgi.py +++ b/Lib/test/test_cgi.py @@ -553,9 +553,10 @@ def test_parse_header(self): ("form-data", {"name": "files", "filename": 'fo"o;bar'})) def test_all(self): - blacklist = {"logfile", "logfp", "initlog", "dolog", "nolog", - "closelog", "log", "maxlen", "valid_boundary"} - support.check__all__(self, cgi, blacklist=blacklist) + not_exported = { + "logfile", "logfp", "initlog", "dolog", "nolog", "closelog", "log", + "maxlen", "valid_boundary"} + support.check__all__(self, cgi, not_exported=not_exported) BOUNDARY = "---------------------------721837373350705526688164684" diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index 230ffc1ccf81a..80a9f179ee2bb 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -2128,8 +2128,7 @@ def test_instance_assignment(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {"Error"} - support.check__all__(self, configparser, blacklist=blacklist) + support.check__all__(self, configparser, not_exported={"Error"}) if __name__ == '__main__': diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py index 65feb3aadedd6..39658f22aa18c 100644 --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -1107,10 +1107,11 @@ def testTimeoutDirectAccess(self): class MiscTestCase(TestCase): def test__all__(self): - blacklist = {'MSG_OOB', 'FTP_PORT', 'MAXLINE', 'CRLF', 'B_CRLF', - 'Error', 'parse150', 'parse227', 'parse229', 'parse257', - 'print_line', 'ftpcp', 'test'} - support.check__all__(self, ftplib, blacklist=blacklist) + not_exported = { + 'MSG_OOB', 'FTP_PORT', 'MAXLINE', 'CRLF', 'B_CRLF', 'Error', + 'parse150', 'parse227', 'parse229', 'parse257', 'print_line', + 'ftpcp', 'test'} + support.check__all__(self, ftplib, not_exported=not_exported) def test_main(): diff --git a/Lib/test/test_gettext.py b/Lib/test/test_gettext.py index df9eae39eac3e..575914d62a0bc 100644 --- a/Lib/test/test_gettext.py +++ b/Lib/test/test_gettext.py @@ -820,8 +820,8 @@ def test_cache(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {'c2py', 'ENOENT'} - support.check__all__(self, gettext, blacklist=blacklist) + support.check__all__(self, gettext, + not_exported={'c2py', 'ENOENT'}) if __name__ == '__main__': diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index d8b3727eb1185..00a4825d6da88 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5363,12 +5363,12 @@ def test_basic(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {'logThreads', 'logMultiprocessing', - 'logProcesses', 'currentframe', - 'PercentStyle', 'StrFormatStyle', 'StringTemplateStyle', - 'Filterer', 'PlaceHolder', 'Manager', 'RootLogger', - 'root', 'threading'} - support.check__all__(self, logging, blacklist=blacklist) + not_exported = { + 'logThreads', 'logMultiprocessing', 'logProcesses', 'currentframe', + 'PercentStyle', 'StrFormatStyle', 'StringTemplateStyle', + 'Filterer', 'PlaceHolder', 'Manager', 'RootLogger', 'root', + 'threading'} + support.check__all__(self, logging, not_exported=not_exported) # Set the locale to the platform-dependent default. I have no idea diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py index 346c9f1c559cd..5924f244f39f8 100644 --- a/Lib/test/test_mailbox.py +++ b/Lib/test/test_mailbox.py @@ -2296,8 +2296,8 @@ def test_nonempty_maildir_both(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {"linesep", "fcntl"} - support.check__all__(self, mailbox, blacklist=blacklist) + support.check__all__(self, mailbox, + not_exported={"linesep", "fcntl"}) def test_main(): diff --git a/Lib/test/test_optparse.py b/Lib/test/test_optparse.py index ed65b7798e3d2..1ed6bf9f919a2 100644 --- a/Lib/test/test_optparse.py +++ b/Lib/test/test_optparse.py @@ -1652,8 +1652,8 @@ def test_numeric_options(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {'check_builtin', 'AmbiguousOptionError', 'NO_DEFAULT'} - support.check__all__(self, optparse, blacklist=blacklist) + not_exported = {'check_builtin', 'AmbiguousOptionError', 'NO_DEFAULT'} + support.check__all__(self, optparse, not_exported=not_exported) def test_main(): diff --git a/Lib/test/test_pickletools.py b/Lib/test/test_pickletools.py index 8cc6ca58cd04b..f5e9ae41c3c1a 100644 --- a/Lib/test/test_pickletools.py +++ b/Lib/test/test_pickletools.py @@ -63,34 +63,35 @@ def test_optimize_binput_and_memoize(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {'bytes_types', - 'UP_TO_NEWLINE', 'TAKEN_FROM_ARGUMENT1', - 'TAKEN_FROM_ARGUMENT4', 'TAKEN_FROM_ARGUMENT4U', - 'TAKEN_FROM_ARGUMENT8U', 'ArgumentDescriptor', - 'read_uint1', 'read_uint2', 'read_int4', 'read_uint4', - 'read_uint8', 'read_stringnl', 'read_stringnl_noescape', - 'read_stringnl_noescape_pair', 'read_string1', - 'read_string4', 'read_bytes1', 'read_bytes4', - 'read_bytes8', 'read_bytearray8', 'read_unicodestringnl', - 'read_unicodestring1', 'read_unicodestring4', - 'read_unicodestring8', 'read_decimalnl_short', - 'read_decimalnl_long', 'read_floatnl', 'read_float8', - 'read_long1', 'read_long4', - 'uint1', 'uint2', 'int4', 'uint4', 'uint8', 'stringnl', - 'stringnl_noescape', 'stringnl_noescape_pair', 'string1', - 'string4', 'bytes1', 'bytes4', 'bytes8', 'bytearray8', - 'unicodestringnl', 'unicodestring1', 'unicodestring4', - 'unicodestring8', 'decimalnl_short', 'decimalnl_long', - 'floatnl', 'float8', 'long1', 'long4', - 'StackObject', - 'pyint', 'pylong', 'pyinteger_or_bool', 'pybool', 'pyfloat', - 'pybytes_or_str', 'pystring', 'pybytes', 'pybytearray', - 'pyunicode', 'pynone', 'pytuple', 'pylist', 'pydict', - 'pyset', 'pyfrozenset', 'pybuffer', 'anyobject', - 'markobject', 'stackslice', 'OpcodeInfo', 'opcodes', - 'code2op', - } - support.check__all__(self, pickletools, blacklist=blacklist) + not_exported = { + 'bytes_types', + 'UP_TO_NEWLINE', 'TAKEN_FROM_ARGUMENT1', + 'TAKEN_FROM_ARGUMENT4', 'TAKEN_FROM_ARGUMENT4U', + 'TAKEN_FROM_ARGUMENT8U', 'ArgumentDescriptor', + 'read_uint1', 'read_uint2', 'read_int4', 'read_uint4', + 'read_uint8', 'read_stringnl', 'read_stringnl_noescape', + 'read_stringnl_noescape_pair', 'read_string1', + 'read_string4', 'read_bytes1', 'read_bytes4', + 'read_bytes8', 'read_bytearray8', 'read_unicodestringnl', + 'read_unicodestring1', 'read_unicodestring4', + 'read_unicodestring8', 'read_decimalnl_short', + 'read_decimalnl_long', 'read_floatnl', 'read_float8', + 'read_long1', 'read_long4', + 'uint1', 'uint2', 'int4', 'uint4', 'uint8', 'stringnl', + 'stringnl_noescape', 'stringnl_noescape_pair', 'string1', + 'string4', 'bytes1', 'bytes4', 'bytes8', 'bytearray8', + 'unicodestringnl', 'unicodestring1', 'unicodestring4', + 'unicodestring8', 'decimalnl_short', 'decimalnl_long', + 'floatnl', 'float8', 'long1', 'long4', + 'StackObject', + 'pyint', 'pylong', 'pyinteger_or_bool', 'pybool', 'pyfloat', + 'pybytes_or_str', 'pystring', 'pybytes', 'pybytearray', + 'pyunicode', 'pynone', 'pytuple', 'pylist', 'pydict', + 'pyset', 'pyfrozenset', 'pybuffer', 'anyobject', + 'markobject', 'stackslice', 'OpcodeInfo', 'opcodes', + 'code2op', + } + support.check__all__(self, pickletools, not_exported=not_exported) def test_main(): diff --git a/Lib/test/test_plistlib.py b/Lib/test/test_plistlib.py index e5038d2e7f10a..e5c9b5b6b2cfe 100644 --- a/Lib/test/test_plistlib.py +++ b/Lib/test/test_plistlib.py @@ -672,8 +672,8 @@ def test_keyed_archive_data(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {"PlistFormat", "PLISTHEADER"} - support.check__all__(self, plistlib, blacklist=blacklist) + not_exported = {"PlistFormat", "PLISTHEADER"} + support.check__all__(self, plistlib, not_exported=not_exported) if __name__ == '__main__': diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py index d5d5abfcf370a..6303192d1b26c 100644 --- a/Lib/test/test_smtpd.py +++ b/Lib/test/test_smtpd.py @@ -1003,12 +1003,11 @@ def test_multiple_emails_with_extended_command_length(self): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = { + not_exported = { "program", "Devnull", "DEBUGSTREAM", "NEWLINE", "COMMASPACE", "DATA_SIZE_DEFAULT", "usage", "Options", "parseargs", - } - support.check__all__(self, smtpd, blacklist=blacklist) + support.check__all__(self, smtpd, not_exported=not_exported) if __name__ == "__main__": diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py index b268511844b82..71a66c27aa21d 100644 --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -391,14 +391,14 @@ def test_detect_api_mismatch__ignore(self): def test_check__all__(self): extra = {'tempdir'} - blacklist = {'template'} + not_exported = {'template'} support.check__all__(self, tempfile, extra=extra, - blacklist=blacklist) + not_exported=not_exported) extra = {'TextTestResult', 'installHandler'} - blacklist = {'load_tests', "TestProgram", "BaseTestSuite"} + not_exported = {'load_tests', "TestProgram", "BaseTestSuite"} support.check__all__(self, unittest, @@ -407,7 +407,7 @@ def test_check__all__(self): "unittest.main", "unittest.runner", "unittest.signals", "unittest.async_case"), extra=extra, - blacklist=blacklist) + not_exported=not_exported) self.assertRaises(AssertionError, support.check__all__, self, unittest) diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py index 4bf7248767c4b..4ef20db097163 100644 --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -2257,22 +2257,19 @@ def test_number_field_limits(self): tarfile.itn(0x10000000000, 6, tarfile.GNU_FORMAT) def test__all__(self): - blacklist = {'version', 'grp', 'pwd', 'symlink_exception', - 'NUL', 'BLOCKSIZE', 'RECORDSIZE', 'GNU_MAGIC', - 'POSIX_MAGIC', 'LENGTH_NAME', 'LENGTH_LINK', - 'LENGTH_PREFIX', 'REGTYPE', 'AREGTYPE', 'LNKTYPE', - 'SYMTYPE', 'CHRTYPE', 'BLKTYPE', 'DIRTYPE', 'FIFOTYPE', - 'CONTTYPE', 'GNUTYPE_LONGNAME', 'GNUTYPE_LONGLINK', - 'GNUTYPE_SPARSE', 'XHDTYPE', 'XGLTYPE', 'SOLARIS_XHDTYPE', - 'SUPPORTED_TYPES', 'REGULAR_TYPES', 'GNU_TYPES', - 'PAX_FIELDS', 'PAX_NAME_FIELDS', 'PAX_NUMBER_FIELDS', - 'stn', 'nts', 'nti', 'itn', 'calc_chksums', 'copyfileobj', - 'filemode', - 'EmptyHeaderError', 'TruncatedHeaderError', - 'EOFHeaderError', 'InvalidHeaderError', - 'SubsequentHeaderError', 'ExFileObject', - 'main'} - support.check__all__(self, tarfile, blacklist=blacklist) + not_exported = { + 'version', 'grp', 'pwd', 'symlink_exception', 'NUL', 'BLOCKSIZE', + 'RECORDSIZE', 'GNU_MAGIC', 'POSIX_MAGIC', 'LENGTH_NAME', + 'LENGTH_LINK', 'LENGTH_PREFIX', 'REGTYPE', 'AREGTYPE', 'LNKTYPE', + 'SYMTYPE', 'CHRTYPE', 'BLKTYPE', 'DIRTYPE', 'FIFOTYPE', 'CONTTYPE', + 'GNUTYPE_LONGNAME', 'GNUTYPE_LONGLINK', 'GNUTYPE_SPARSE', + 'XHDTYPE', 'XGLTYPE', 'SOLARIS_XHDTYPE', 'SUPPORTED_TYPES', + 'REGULAR_TYPES', 'GNU_TYPES', 'PAX_FIELDS', 'PAX_NAME_FIELDS', + 'PAX_NUMBER_FIELDS', 'stn', 'nts', 'nti', 'itn', 'calc_chksums', + 'copyfileobj', 'filemode', 'EmptyHeaderError', + 'TruncatedHeaderError', 'EOFHeaderError', 'InvalidHeaderError', + 'SubsequentHeaderError', 'ExFileObject', 'main'} + support.check__all__(self, tarfile, not_exported=not_exported) class CommandLineTest(unittest.TestCase): diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index 47e131ae27a14..d02d3b346f9cc 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -1365,9 +1365,9 @@ class BarrierTests(lock_tests.BarrierTests): class MiscTestCase(unittest.TestCase): def test__all__(self): extra = {"ThreadError"} - blacklist = {'currentThread', 'activeCount'} + not_exported = {'currentThread', 'activeCount'} support.check__all__(self, threading, ('threading', '_thread'), - extra=extra, blacklist=blacklist) + extra=extra, not_exported=not_exported) class InterruptMainTests(unittest.TestCase): diff --git a/Lib/test/test_wave.py b/Lib/test/test_wave.py index eb231cb19c6d0..f85e40b31d010 100644 --- a/Lib/test/test_wave.py +++ b/Lib/test/test_wave.py @@ -107,8 +107,7 @@ class WavePCM32Test(WaveTest, unittest.TestCase): class MiscTestCase(unittest.TestCase): def test__all__(self): - blacklist = {'WAVE_FORMAT_PCM'} - support.check__all__(self, wave, blacklist=blacklist) + support.check__all__(self, wave, not_exported={'WAVE_FORMAT_PCM'}) class WaveLowLevelTest(unittest.TestCase): diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index c7d446185cfe4..d22c35d92c4e3 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -128,7 +128,7 @@ def test_sanity(self): def test_all(self): names = ("xml.etree.ElementTree", "_elementtree") - support.check__all__(self, ET, names, blacklist=("HTML_EMPTY",)) + support.check__all__(self, ET, names, not_exported=("HTML_EMPTY",)) def serialize(elem, to_string=True, encoding='unicode', **options): diff --git a/Misc/NEWS.d/next/Tests/2020-08-11-14-59-13.bpo-41521.w2UYK7.rst b/Misc/NEWS.d/next/Tests/2020-08-11-14-59-13.bpo-41521.w2UYK7.rst new file mode 100644 index 0000000000000..658372b1a7f22 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2020-08-11-14-59-13.bpo-41521.w2UYK7.rst @@ -0,0 +1,2 @@ +:mod:`test.support`: Rename ``blacklist`` parameter of +:func:`~test.support.check__all__` to ``not_exported``. From webhook-mailer at python.org Mon Aug 17 02:41:47 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Mon, 17 Aug 2020 06:41:47 -0000 Subject: [Python-checkins] bpo-40204: Fix duplicated productionlist names in the doc (GH-21900) Message-ID: https://github.com/python/cpython/commit/1abeda80f760134b4233608e2c288790f955b95a commit: 1abeda80f760134b4233608e2c288790f955b95a branch: master author: Victor Stinner committer: GitHub date: 2020-08-17T08:41:42+02:00 summary: bpo-40204: Fix duplicated productionlist names in the doc (GH-21900) Sphinx 3 disallows having more than one productionlist markup with the same name. Simply remove names in this case, since names are not shown anyway. For example, fix the Sphinx 3 warning: Doc/reference/introduction.rst:96: duplicate token description of *:name, other instance in reference/expressions files: M Doc/library/string.rst M Doc/reference/introduction.rst diff --git a/Doc/library/string.rst b/Doc/library/string.rst index fa906f799c108..62e86d6dd9706 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -308,7 +308,7 @@ non-empty format specification typically modifies the result. The general form of a *standard format specifier* is: -.. productionlist:: sf +.. productionlist:: format_spec: [[`fill`]`align`][`sign`][#][0][`width`][`grouping_option`][.`precision`][`type`] fill: align: "<" | ">" | "=" | "^" diff --git a/Doc/reference/introduction.rst b/Doc/reference/introduction.rst index bb7e3906dba60..62480bd7dd9a6 100644 --- a/Doc/reference/introduction.rst +++ b/Doc/reference/introduction.rst @@ -93,7 +93,7 @@ Notation The descriptions of lexical analysis and syntax use a modified BNF grammar notation. This uses the following style of definition: -.. productionlist:: * +.. productionlist:: name: `lc_letter` (`lc_letter` | "_")* lc_letter: "a"..."z" From webhook-mailer at python.org Mon Aug 17 09:39:01 2020 From: webhook-mailer at python.org (Allen) Date: Mon, 17 Aug 2020 13:39:01 -0000 Subject: [Python-checkins] Fix typo in message from assert statement (GH-21283) Message-ID: https://github.com/python/cpython/commit/99c0ee3c893bb99fd98a97084fc386ce2911eb64 commit: 99c0ee3c893bb99fd98a97084fc386ce2911eb64 branch: master author: Allen <64019758+aboddie at users.noreply.github.com> committer: GitHub date: 2020-08-17T19:08:55+05:30 summary: Fix typo in message from assert statement (GH-21283) The error message was missing space between the action "acquire" and "_wait_semaphore" which is an attribute for instances of Condition. files: M Lib/multiprocessing/synchronize.py diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py index 4fcbefc8bbefd..d0be48f1fd7a8 100644 --- a/Lib/multiprocessing/synchronize.py +++ b/Lib/multiprocessing/synchronize.py @@ -270,7 +270,7 @@ def wait(self, timeout=None): def notify(self, n=1): assert self._lock._semlock._is_mine(), 'lock is not owned' assert not self._wait_semaphore.acquire( - False), ('notify: Should not have been able to acquire' + False), ('notify: Should not have been able to acquire ' + '_wait_semaphore') # to take account of timeouts since last notify*() we subtract From webhook-mailer at python.org Mon Aug 17 10:20:05 2020 From: webhook-mailer at python.org (James Weaver) Date: Mon, 17 Aug 2020 14:20:05 -0000 Subject: [Python-checkins] bpo-40782: Change asyncio.AbstractEventLoop.run_in_executor to be a method not a coroutine (GH-21852) Message-ID: https://github.com/python/cpython/commit/29f84294d88ec493c2de9d6e8dbc12fae3778771 commit: 29f84294d88ec493c2de9d6e8dbc12fae3778771 branch: master author: James Weaver committer: GitHub date: 2020-08-17T07:19:46-07:00 summary: bpo-40782: Change asyncio.AbstractEventLoop.run_in_executor to be a method not a coroutine (GH-21852) asyncio.AbstractEventLoop.run_in_executor should be a method that returns an asyncio Future, not an async method. This matches the concrete implementations, and the documentation better. files: A Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst M Lib/asyncio/events.py diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index 70017cb86a059..0dce87b8ecc58 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -283,7 +283,7 @@ def create_task(self, coro, *, name=None): def call_soon_threadsafe(self, callback, *args): raise NotImplementedError - async def run_in_executor(self, executor, func, *args): + def run_in_executor(self, executor, func, *args): raise NotImplementedError def set_default_executor(self, executor): diff --git a/Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst b/Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst new file mode 100644 index 0000000000000..d4c7e0e2419df --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst @@ -0,0 +1 @@ +Change the method asyncio.AbstractEventLoop.run_in_executor to not be a coroutine. \ No newline at end of file From webhook-mailer at python.org Mon Aug 17 10:37:32 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 17 Aug 2020 14:37:32 -0000 Subject: [Python-checkins] bpo-40782: Change asyncio.AbstractEventLoop.run_in_executor to be a method not a coroutine (GH-21852) Message-ID: https://github.com/python/cpython/commit/1baa8b14ee23ef3040923f53565c8d1bafd28117 commit: 1baa8b14ee23ef3040923f53565c8d1bafd28117 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-17T07:37:12-07:00 summary: bpo-40782: Change asyncio.AbstractEventLoop.run_in_executor to be a method not a coroutine (GH-21852) asyncio.AbstractEventLoop.run_in_executor should be a method that returns an asyncio Future, not an async method. This matches the concrete implementations, and the documentation better. (cherry picked from commit 29f84294d88ec493c2de9d6e8dbc12fae3778771) Co-authored-by: James Weaver files: A Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst M Lib/asyncio/events.py diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index ca08663a5b358..353a9f5af0c49 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -280,7 +280,7 @@ def create_task(self, coro, *, name=None): def call_soon_threadsafe(self, callback, *args): raise NotImplementedError - async def run_in_executor(self, executor, func, *args): + def run_in_executor(self, executor, func, *args): raise NotImplementedError def set_default_executor(self, executor): diff --git a/Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst b/Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst new file mode 100644 index 0000000000000..d4c7e0e2419df --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-13-08-07-25.bpo-40782.aGZqmB.rst @@ -0,0 +1 @@ +Change the method asyncio.AbstractEventLoop.run_in_executor to not be a coroutine. \ No newline at end of file From webhook-mailer at python.org Mon Aug 17 11:32:34 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Mon, 17 Aug 2020 15:32:34 -0000 Subject: [Python-checkins] Fix typo in message from assert statement (GH-21283) Message-ID: https://github.com/python/cpython/commit/2bcd0fe7a5d1a3c3dd99e7e067239a514a780402 commit: 2bcd0fe7a5d1a3c3dd99e7e067239a514a780402 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-17T08:32:11-07:00 summary: Fix typo in message from assert statement (GH-21283) The error message was missing space between the action "acquire" and "_wait_semaphore" which is an attribute for instances of Condition. (cherry picked from commit 99c0ee3c893bb99fd98a97084fc386ce2911eb64) Co-authored-by: Allen <64019758+aboddie at users.noreply.github.com> files: M Lib/multiprocessing/synchronize.py diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py index 4fcbefc8bbefd..d0be48f1fd7a8 100644 --- a/Lib/multiprocessing/synchronize.py +++ b/Lib/multiprocessing/synchronize.py @@ -270,7 +270,7 @@ def wait(self, timeout=None): def notify(self, n=1): assert self._lock._semlock._is_mine(), 'lock is not owned' assert not self._wait_semaphore.acquire( - False), ('notify: Should not have been able to acquire' + False), ('notify: Should not have been able to acquire ' + '_wait_semaphore') # to take account of timeouts since last notify*() we subtract From webhook-mailer at python.org Mon Aug 17 16:36:27 2020 From: webhook-mailer at python.org (Hai Shi) Date: Mon, 17 Aug 2020 20:36:27 -0000 Subject: [Python-checkins] bpo-1635741: Explict GC collect after PyInterpreterState_Clear() (GH-21902) Message-ID: https://github.com/python/cpython/commit/8aa163eea6b0fb4693f6c0a314d4f2ccada51d70 commit: 8aa163eea6b0fb4693f6c0a314d4f2ccada51d70 branch: master author: Hai Shi committer: GitHub date: 2020-08-17T22:36:19+02:00 summary: bpo-1635741: Explict GC collect after PyInterpreterState_Clear() (GH-21902) Fix a reference cycle by triggering an explicit GC collection after calling PyInterpreterState_Clear(). files: M Python/pylifecycle.c diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 2d219a4a3a8b0..ab5a6767864dc 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -1286,10 +1286,8 @@ finalize_interp_clear(PyThreadState *tstate) /* Clear interpreter state and all thread states */ PyInterpreterState_Clear(tstate->interp); - /* Trigger a GC collection on subinterpreters*/ - if (!is_main_interp) { - _PyGC_CollectNoFail(); - } + /* Last explicit GC collection */ + _PyGC_CollectNoFail(); /* Clear all loghooks */ /* Both _PySys_Audit function and users still need PyObject, such as tuple. From webhook-mailer at python.org Mon Aug 17 18:25:44 2020 From: webhook-mailer at python.org (Ned Deily) Date: Mon, 17 Aug 2020 22:25:44 -0000 Subject: [Python-checkins] bpo-41100: additional fixes for testing on macOS 11 Big Sur Intel Message-ID: https://github.com/python/cpython/commit/a0ad82959652ff64c99231f457fd740b17330514 commit: a0ad82959652ff64c99231f457fd740b17330514 branch: 3.7 author: Ned Deily committer: Ned Deily date: 2020-08-15T01:08:56-04:00 summary: bpo-41100: additional fixes for testing on macOS 11 Big Sur Intel Note: macOS 11 is not yet released, this release of Python is not fully supported on 11.0, and not all tests pass. files: A Misc/NEWS.d/next/macOS/2020-08-15-00-33-27.bpo-41100.AksBg1.rst M Lib/distutils/tests/test_build_ext.py M Lib/test/test_platform.py M Modules/getpath.c M configure M configure.ac diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py index 88847f9e9aa7b..d0428599a40af 100644 --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -470,7 +470,7 @@ def _try_compile_deployment_target(self, operator, target): # format the target value as defined in the Apple # Availability Macros. We can't use the macro names since # at least one value we test with will not exist yet. - if target[1] < 10: + if target[:2] < (10, 10): # for 10.1 through 10.9.x -> "10n0" target = '%02d%01d0' % target else: diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py index d91e978a79f3f..452a56e8b4e2b 100644 --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -236,6 +236,11 @@ def test_mac_ver(self): fd.close() self.assertFalse(real_ver is None) result_list = res[0].split('.') + # macOS 11.0 (Big Sur) may report its version number + # as 10.16 if the executable is built with an older + # SDK target but sw_vers reports 11.0. + if result_list == ['10', '16']: + result_list = ['11', '0'] expect_list = real_ver.split('.') len_diff = len(result_list) - len(expect_list) # On Snow Leopard, sw_vers reports 10.6.0 as 10.6 diff --git a/Misc/NEWS.d/next/macOS/2020-08-15-00-33-27.bpo-41100.AksBg1.rst b/Misc/NEWS.d/next/macOS/2020-08-15-00-33-27.bpo-41100.AksBg1.rst new file mode 100644 index 0000000000000..b6e12dc340440 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2020-08-15-00-33-27.bpo-41100.AksBg1.rst @@ -0,0 +1,3 @@ +Additional fixes for testing on macOS 11 Big Sur Intel. Note: macOS 11 is +not yet released, this release of Python is not fully supported on 11.0, and +not all tests pass. diff --git a/Modules/getpath.c b/Modules/getpath.c index ba8d74b4a0427..35f9e31d1ed4d 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -566,11 +566,7 @@ calculate_program_full_path(const _PyCoreConfig *core_config, memset(program_full_path, 0, sizeof(program_full_path)); #ifdef __APPLE__ -#if MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_4 uint32_t nsexeclength = MAXPATHLEN; -#else - unsigned long nsexeclength = MAXPATHLEN; -#endif char execpath[MAXPATHLEN+1]; #endif diff --git a/configure b/configure index c807c98e568f8..829dd69bb8b30 100755 --- a/configure +++ b/configure @@ -9251,6 +9251,9 @@ fi ppc) MACOSX_DEFAULT_ARCH="ppc64" ;; + arm64) + MACOSX_DEFAULT_ARCH="arm64" + ;; *) as_fn_error $? "Unexpected output of 'arch' on OSX" "$LINENO" 5 ;; diff --git a/configure.ac b/configure.ac index 805c0bba08deb..f1cc8e9bcb9ed 100644 --- a/configure.ac +++ b/configure.ac @@ -2456,6 +2456,9 @@ case $ac_sys_system/$ac_sys_release in ppc) MACOSX_DEFAULT_ARCH="ppc64" ;; + arm64) + MACOSX_DEFAULT_ARCH="arm64" + ;; *) AC_MSG_ERROR([Unexpected output of 'arch' on OSX]) ;; From webhook-mailer at python.org Mon Aug 17 18:26:03 2020 From: webhook-mailer at python.org (Ned Deily) Date: Mon, 17 Aug 2020 22:26:03 -0000 Subject: [Python-checkins] 3.6.12 Message-ID: https://github.com/python/cpython/commit/c0a9afe2ac1820409e6173bd1893ebee2cf50270 commit: c0a9afe2ac1820409e6173bd1893ebee2cf50270 branch: 3.6 author: Ned Deily committer: Ned Deily date: 2020-08-15T02:43:26-04:00 summary: 3.6.12 files: A Misc/NEWS.d/3.6.12.rst D Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst D Misc/NEWS.d/next/Library/2020-07-13-15-06-35.bpo-41288.8mn5P-.rst D Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst D Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst D Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst M Include/patchlevel.h M Lib/pydoc_data/topics.py M README.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index 1f4195058aeb1..7478a6b530893 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -18,12 +18,12 @@ /*--start constants--*/ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 6 -#define PY_MICRO_VERSION 11 +#define PY_MICRO_VERSION 12 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.6.11+" +#define PY_VERSION "3.6.12" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 69bf815cab361..4b53f76969f8a 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Wed Jun 17 06:55:37 2020 +# Autogenerated by Sphinx on Sat Aug 15 02:33:47 2020 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' diff --git a/Misc/NEWS.d/3.6.12.rst b/Misc/NEWS.d/3.6.12.rst new file mode 100644 index 0000000000000..277b3fe13e350 --- /dev/null +++ b/Misc/NEWS.d/3.6.12.rst @@ -0,0 +1,51 @@ +.. bpo: 29778 +.. date: 2020-07-03-17-21-37 +.. nonce: cR_fGS +.. release date: 2020-08-15 +.. section: Security + +Ensure :file:`python3.dll` is loaded from correct locations when Python is +embedded (CVE-2020-15523). + +.. + +.. bpo: 41004 +.. date: 2020-06-29-16-02-29 +.. nonce: ovF0KZ +.. section: Security + +CVE-2020-14422: The __hash__() methods of ipaddress.IPv4Interface and +ipaddress.IPv6Interface incorrectly generated constant hash values of 32 and +128 respectively. This resulted in always causing hash collisions. The fix +uses hash() to generate hash values for the tuple of (address, mask length, +network address). + +.. + +.. bpo: 39603 +.. date: 2020-02-12-14-17-39 +.. nonce: Gt3RSg +.. section: Security + +Prevent http header injection by rejecting control characters in +http.client.putrequest(...). + +.. + +.. bpo: 41288 +.. date: 2020-07-13-15-06-35 +.. nonce: 8mn5P- +.. section: Library + +Unpickling invalid NEWOBJ_EX opcode with the C implementation raises now +UnpicklingError instead of crashing. + +.. + +.. bpo: 39017 +.. date: 2020-07-12-22-16-58 +.. nonce: x3Cg-9 +.. section: Library + +Avoid infinite loop when reading specially crafted TAR files using the +tarfile module (CVE-2019-20907). diff --git a/Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst b/Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst deleted file mode 100644 index ad26676f8b856..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst +++ /dev/null @@ -1 +0,0 @@ -Avoid infinite loop when reading specially crafted TAR files using the tarfile module (CVE-2019-20907). diff --git a/Misc/NEWS.d/next/Library/2020-07-13-15-06-35.bpo-41288.8mn5P-.rst b/Misc/NEWS.d/next/Library/2020-07-13-15-06-35.bpo-41288.8mn5P-.rst deleted file mode 100644 index 3c3adbabf16ff..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-13-15-06-35.bpo-41288.8mn5P-.rst +++ /dev/null @@ -1,2 +0,0 @@ -Unpickling invalid NEWOBJ_EX opcode with the C implementation raises now -UnpicklingError instead of crashing. diff --git a/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst b/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst deleted file mode 100644 index 990affc3edd9d..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-02-12-14-17-39.bpo-39603.Gt3RSg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Prevent http header injection by rejecting control characters in -http.client.putrequest(...). diff --git a/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst b/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst deleted file mode 100644 index f5a9db52fff52..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst +++ /dev/null @@ -1 +0,0 @@ -CVE-2020-14422: The __hash__() methods of ipaddress.IPv4Interface and ipaddress.IPv6Interface incorrectly generated constant hash values of 32 and 128 respectively. This resulted in always causing hash collisions. The fix uses hash() to generate hash values for the tuple of (address, mask length, network address). diff --git a/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst b/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst deleted file mode 100644 index 998ffb1ee6667..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst +++ /dev/null @@ -1,2 +0,0 @@ -Ensure :file:`python3.dll` is loaded from correct locations when Python is -embedded (CVE-2020-15523). diff --git a/README.rst b/README.rst index c6602bb842914..fdcbb3eeb4a60 100644 --- a/README.rst +++ b/README.rst @@ -1,5 +1,5 @@ -This is Python version 3.6.11+ -============================== +This is Python version 3.6.12 +============================= .. image:: https://travis-ci.org/python/cpython.svg?branch=3.6 :alt: CPython build status on Travis CI From webhook-mailer at python.org Mon Aug 17 18:40:17 2020 From: webhook-mailer at python.org (Paul Ganssle) Date: Mon, 17 Aug 2020 22:40:17 -0000 Subject: [Python-checkins] bpo-41568: Fix refleaks in zoneinfo subclasses (GH-21907) Message-ID: https://github.com/python/cpython/commit/c3dd7e45cc5d36bbe2295c2840faabb5c75d83e4 commit: c3dd7e45cc5d36bbe2295c2840faabb5c75d83e4 branch: master author: Paul Ganssle committer: GitHub date: 2020-08-17T23:40:07+01:00 summary: bpo-41568: Fix refleaks in zoneinfo subclasses (GH-21907) * Fix refleak in C module __init_subclass__ This was leaking a reference to the weak cache dictionary for every ZoneInfo subclass created. * Fix refleak in ZoneInfo subclass's clear_cache The previous version of the code accidentally cleared the global ZONEINFO_STRONG_CACHE variable (and inducing `ZoneInfo` to create a new strong cache) on calls to a subclass's `clear_cache()`. This would not affect guaranteed behavior, but it's still not the right thing to do (and it caused reference leaks). files: M Modules/_zoneinfo.c diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index 12b3969959bac..2cee65fac6dd0 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -412,7 +412,6 @@ zoneinfo_clear_cache(PyObject *cls, PyObject *args, PyObject *kwargs) } clear_strong_cache(type); - ZONEINFO_STRONG_CACHE = NULL; } else { PyObject *item = NULL; @@ -2471,6 +2470,7 @@ clear_strong_cache(const PyTypeObject *const type) } strong_cache_free(ZONEINFO_STRONG_CACHE); + ZONEINFO_STRONG_CACHE = NULL; } static PyObject * @@ -2525,6 +2525,7 @@ zoneinfo_init_subclass(PyTypeObject *cls, PyObject *args, PyObject **kwargs) } PyObject_SetAttrString((PyObject *)cls, "_weak_cache", weak_cache); + Py_DECREF(weak_cache); Py_RETURN_NONE; } @@ -2616,8 +2617,7 @@ module_free() Py_CLEAR(ZONEINFO_WEAK_CACHE); } - strong_cache_free(ZONEINFO_STRONG_CACHE); - ZONEINFO_STRONG_CACHE = NULL; + clear_strong_cache(&PyZoneInfo_ZoneInfoType); } static int From webhook-mailer at python.org Wed Aug 19 07:30:01 2020 From: webhook-mailer at python.org (Denis Ovsienko) Date: Wed, 19 Aug 2020 11:30:01 -0000 Subject: [Python-checkins] Fix grammar in Doc/tutorial/controlflow.rst (GH-21885) Message-ID: https://github.com/python/cpython/commit/0be7c216e16f0d459f1c8f6209734c9b2b82fbd4 commit: 0be7c216e16f0d459f1c8f6209734c9b2b82fbd4 branch: master author: Denis Ovsienko committer: GitHub date: 2020-08-19T04:29:47-07:00 summary: Fix grammar in Doc/tutorial/controlflow.rst (GH-21885) Automerge-Triggered-By: @csabella files: M Doc/tutorial/controlflow.rst diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index 5d5b01d813277..5d24a19cfc079 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -661,7 +661,7 @@ Finally, consider this function definition which has a potential collision betwe return 'name' in kwds There is no possible call that will make it return ``True`` as the keyword ``'name'`` -will always to bind to the first parameter. For example:: +will always bind to the first parameter. For example:: >>> foo(1, **{'name': 2}) Traceback (most recent call last): From webhook-mailer at python.org Wed Aug 19 15:34:58 2020 From: webhook-mailer at python.org (Frank Dana) Date: Wed, 19 Aug 2020 19:34:58 -0000 Subject: [Python-checkins] [3.8] venv: Update Aspen URL in 'activate' script comment (GH-21797) Message-ID: https://github.com/python/cpython/commit/34889a5785d4cc34e06b331dc36d479404a34df7 commit: 34889a5785d4cc34e06b331dc36d479404a34df7 branch: 3.8 author: Frank Dana committer: GitHub date: 2020-08-19T12:34:50-07:00 summary: [3.8] venv: Update Aspen URL in 'activate' script comment (GH-21797) A comment in the venv `activate` script (as well as `activate.csh` and `activate.fish`) referencing Aspen magic directories lists a "further information" URL for Aspen at the zetadev.com website. zetadev.com changed ownership in 2019, and now redirects to a server in China with an expired security certificate. Out of an abundance of caution, while not changing the _code_ for the activate scripts, this PR updates the URL in those comments to reference Aspen's new documentation home at https://aspen.io/. No issue created, as I suspect this falls within the definition of a "trivial" change. Please let me know if I'm wrong about that, and I'll open the necessary issue(s). While filed against the 3.8 branch, strictly speaking this is not a backported PR. The comment in question was entirely removed from the script between Python 3.8 and 3.9. (IMHO this _should_ probably be backported to 3.7 and 3.6, as well. I'll happily file those PRs if needed.) Automerge-Triggered-By: @vsajip files: M Lib/venv/scripts/common/activate M Lib/venv/scripts/posix/activate.csh M Lib/venv/scripts/posix/activate.fish diff --git a/Lib/venv/scripts/common/activate b/Lib/venv/scripts/common/activate index b9d498fb2ef04..5e7ac1746813e 100644 --- a/Lib/venv/scripts/common/activate +++ b/Lib/venv/scripts/common/activate @@ -59,7 +59,7 @@ if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then else if [ "`basename \"$VIRTUAL_ENV\"`" = "__" ] ; then # special case for Aspen magic directories - # see http://www.zetadev.com/software/aspen/ + # see https://aspen.io/ PS1="[`basename \`dirname \"$VIRTUAL_ENV\"\``] $PS1" else PS1="(`basename \"$VIRTUAL_ENV\"`)$PS1" diff --git a/Lib/venv/scripts/posix/activate.csh b/Lib/venv/scripts/posix/activate.csh index b0c7028a92955..0f39ee8cfd654 100644 --- a/Lib/venv/scripts/posix/activate.csh +++ b/Lib/venv/scripts/posix/activate.csh @@ -22,7 +22,7 @@ if (! "$?VIRTUAL_ENV_DISABLE_PROMPT") then else if (`basename "VIRTUAL_ENV"` == "__") then # special case for Aspen magic directories - # see http://www.zetadev.com/software/aspen/ + # see https://aspen.io/ set env_name = `basename \`dirname "$VIRTUAL_ENV"\`` else set env_name = `basename "$VIRTUAL_ENV"` diff --git a/Lib/venv/scripts/posix/activate.fish b/Lib/venv/scripts/posix/activate.fish index b40105825eadb..03e893f8eb5d9 100644 --- a/Lib/venv/scripts/posix/activate.fish +++ b/Lib/venv/scripts/posix/activate.fish @@ -59,7 +59,7 @@ if test -z "$VIRTUAL_ENV_DISABLE_PROMPT" set -l _checkbase (basename "$VIRTUAL_ENV") if test $_checkbase = "__" # special case for Aspen magic directories - # see http://www.zetadev.com/software/aspen/ + # see https://aspen.io/ printf "%s[%s]%s " (set_color -b blue white) (basename (dirname "$VIRTUAL_ENV")) (set_color normal) else printf "%s(%s)%s" (set_color -b blue white) (basename "$VIRTUAL_ENV") (set_color normal) From webhook-mailer at python.org Thu Aug 20 06:30:45 2020 From: webhook-mailer at python.org (Sydney Pemberton) Date: Thu, 20 Aug 2020 10:30:45 -0000 Subject: [Python-checkins] bpo-40994: Ungroup items in collections.abc documentation for improved clarity (GH-21880) Message-ID: https://github.com/python/cpython/commit/2ce39631f679e14132a54dc90ce764259d26e166 commit: 2ce39631f679e14132a54dc90ce764259d26e166 branch: master author: Sydney Pemberton <46042811+sydneypemberton1986 at users.noreply.github.com> committer: GitHub date: 2020-08-20T03:30:21-07:00 summary: bpo-40994: Ungroup items in collections.abc documentation for improved clarity (GH-21880) Use a less surprising document structure. Automerge-Triggered-By: @csabella files: M Doc/library/collections.abc.rst diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index dc7ae30b6d2fa..db0e25bb0772e 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -98,12 +98,20 @@ ABC Inherits from Abstract Methods Mixin .. class:: Container - Hashable - Sized - Callable - ABCs for classes that provide respectively the methods :meth:`__contains__`, - :meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`. + ABC for classes that provide the :meth:`__contains__` method. + +.. class:: Hashable + + ABC for classes that provide the :meth:`__hash__` method. + +.. class:: Sized + + ABC for classes that provide the :meth:`__len__` method. + +.. class:: Callable + + ABC for classes that provide the :meth:`__call__` method. .. class:: Iterable From webhook-mailer at python.org Thu Aug 20 07:28:55 2020 From: webhook-mailer at python.org (Victor Stinner) Date: Thu, 20 Aug 2020 11:28:55 -0000 Subject: [Python-checkins] bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901) (GH-21928) Message-ID: https://github.com/python/cpython/commit/7d0fef56d8eaac6309a66cb8c6ba6fd96f8c8a94 commit: 7d0fef56d8eaac6309a66cb8c6ba6fd96f8c8a94 branch: 3.8 author: Victor Stinner committer: GitHub date: 2020-08-20T13:28:49+02:00 summary: bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) (GH-21901) (GH-21928) * bpo-40204: Allow pre-Sphinx 3 syntax in the doc (GH-21844) Enable Sphinx 3.2 "c_allow_pre_v3" option and disable the c_warn_on_allowed_pre_v3 option to make the documentation compatible with Sphinx 2 and Sphinx 3. (cherry picked from commit 423e77d6de497931585d1883805a9e3fa4096b0b) * bpo-40204: Fix Sphinx sytanx in howto/instrumentation.rst (GH-21858) Use generic '.. object::' to declare markers, rather than abusing '.. c:function::' which fails on Sphinx 3. (cherry picked from commit 43577c01a2ab49122db696e9eaec6cb31d11cc81) * bpo-40204: Fix duplicates in the documentation (GH-21857) Fix two Sphinx 3 issues: Doc/c-api/buffer.rst:304: WARNING: Duplicate C declaration, also defined in 'c-api/buffer'. Declaration is 'PyBUF_ND'. Doc/c-api/unicode.rst:1603: WARNING: Duplicate C declaration, also defined in 'c-api/unicode'. Declaration is 'PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors)'. (cherry picked from commit 46d10b1237c67ff8347f533eda6a5468d098f7eb) * bpo-40204: Add :noindex: in the documentation (GH-21859) Add :noindex: to duplicated documentation to fix "duplicate object description" errors. For example, fix this Sphinx 3 issue: Doc/library/configparser.rst:1146: WARNING: duplicate object description of configparser.ConfigParser.optionxform, other instance in library/configparser, use :noindex: for one of them (cherry picked from commit d3ded080482beae578faa704b13534a62d066f9f) * bpo-40204, doc: Fix syntax of C variables (GH-21846) For example, fix the following Sphinx 3 errors: Doc/c-api/buffer.rst:102: WARNING: Error in declarator or parameters Invalid C declaration: Expected identifier in nested name. [error at 5] void \*obj -----^ Doc/c-api/arg.rst:130: WARNING: Unparseable C cross-reference: 'PyObject*' Invalid C declaration: Expected end of definition. [error at 8] PyObject* --------^ The modified documentation is compatible with Sphinx 2 and Sphinx 3. (cherry picked from commit 474652fe9346382dbf793f20b671eb74668bebde) * bpo-40204: Fix reference to terms in the doc (GH-21865) Sphinx 3 requires to refer to terms with the exact case. For example, fix the Sphinx 3 warning: Doc/library/pkgutil.rst:71: WARNING: term Loader not found in case sensitive match.made a reference to loader instead. (cherry picked from commit bb0b08540cc93e56f3f1bde1b39ce086d9e35fe1) * bpo-40204: Fix duplicated productionlist names in the doc (GH-21900) Sphinx 3 disallows having more than one productionlist markup with the same name. Simply remove names in this case, since names are not shown anyway. For example, fix the Sphinx 3 warning: Doc/reference/introduction.rst:96: duplicate token description of *:name, other instance in reference/expressions (cherry picked from commit 1abeda80f760134b4233608e2c288790f955b95a) (cherry picked from commit 8f88190af529543c84d5dc78f19abbfd73335cf4) files: A Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst M Doc/c-api/arg.rst M Doc/c-api/buffer.rst M Doc/c-api/capsule.rst M Doc/c-api/dict.rst M Doc/c-api/exceptions.rst M Doc/c-api/file.rst M Doc/c-api/init.rst M Doc/c-api/intro.rst M Doc/c-api/marshal.rst M Doc/c-api/memory.rst M Doc/c-api/module.rst M Doc/c-api/object.rst M Doc/c-api/structures.rst M Doc/c-api/tuple.rst M Doc/c-api/typeobj.rst M Doc/c-api/unicode.rst M Doc/c-api/veryhigh.rst M Doc/conf.py M Doc/extending/newtypes_tutorial.rst M Doc/glossary.rst M Doc/howto/instrumentation.rst M Doc/library/aifc.rst M Doc/library/collections.abc.rst M Doc/library/concurrent.futures.rst M Doc/library/configparser.rst M Doc/library/difflib.rst M Doc/library/enum.rst M Doc/library/importlib.rst M Doc/library/multiprocessing.rst M Doc/library/pkgutil.rst M Doc/library/socket.rst M Doc/library/string.rst M Doc/library/tarfile.rst M Doc/library/threading.rst M Doc/library/token.rst M Doc/library/turtle.rst M Doc/library/urllib.request.rst M Doc/reference/datamodel.rst M Doc/reference/introduction.rst M Doc/tutorial/classes.rst M Doc/whatsnew/3.2.rst M Doc/whatsnew/3.3.rst M Doc/whatsnew/3.5.rst M Doc/whatsnew/3.6.rst M Doc/whatsnew/3.7.rst diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index b7baad589a72c..a187a8fb81437 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -131,12 +131,12 @@ which disallows mutable objects such as :class:`bytearray`. ``S`` (:class:`bytes`) [PyBytesObject \*] Requires that the Python object is a :class:`bytes` object, without attempting any conversion. Raises :exc:`TypeError` if the object is not - a bytes object. The C variable may also be declared as :c:type:`PyObject\*`. + a bytes object. The C variable may also be declared as :c:type:`PyObject*`. ``Y`` (:class:`bytearray`) [PyByteArrayObject \*] Requires that the Python object is a :class:`bytearray` object, without attempting any conversion. Raises :exc:`TypeError` if the object is not - a :class:`bytearray` object. The C variable may also be declared as :c:type:`PyObject\*`. + a :class:`bytearray` object. The C variable may also be declared as :c:type:`PyObject*`. ``u`` (:class:`str`) [const Py_UNICODE \*] Convert a Python Unicode object to a C pointer to a NUL-terminated buffer of @@ -183,7 +183,7 @@ which disallows mutable objects such as :class:`bytearray`. ``U`` (:class:`str`) [PyObject \*] Requires that the Python object is a Unicode object, without attempting any conversion. Raises :exc:`TypeError` if the object is not a Unicode - object. The C variable may also be declared as :c:type:`PyObject\*`. + object. The C variable may also be declared as :c:type:`PyObject*`. ``w*`` (read-write :term:`bytes-like object`) [Py_buffer] This format accepts any object which implements the read-write buffer @@ -196,10 +196,10 @@ which disallows mutable objects such as :class:`bytearray`. It only works for encoded data without embedded NUL bytes. This format requires two arguments. The first is only used as input, and - must be a :c:type:`const char\*` which points to the name of an encoding as a + must be a :c:type:`const char*` which points to the name of an encoding as a NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used. An exception is raised if the named encoding is not known to Python. The - second argument must be a :c:type:`char\*\*`; the value of the pointer it + second argument must be a :c:type:`char**`; the value of the pointer it references will be set to a buffer with the contents of the argument text. The text will be encoded in the encoding specified by the first argument. @@ -219,10 +219,10 @@ which disallows mutable objects such as :class:`bytearray`. characters. It requires three arguments. The first is only used as input, and must be a - :c:type:`const char\*` which points to the name of an encoding as a + :c:type:`const char*` which points to the name of an encoding as a NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used. An exception is raised if the named encoding is not known to Python. The - second argument must be a :c:type:`char\*\*`; the value of the pointer it + second argument must be a :c:type:`char**`; the value of the pointer it references will be set to a buffer with the contents of the argument text. The text will be encoded in the encoding specified by the first argument. The third argument must be a pointer to an integer; the referenced integer @@ -322,7 +322,7 @@ Other objects ``O!`` (object) [*typeobject*, PyObject \*] Store a Python object in a C object pointer. This is similar to ``O``, but takes two C arguments: the first is the address of a Python type object, the - second is the address of the C variable (of type :c:type:`PyObject\*`) into which + second is the address of the C variable (of type :c:type:`PyObject*`) into which the object pointer is stored. If the Python object does not have the required type, :exc:`TypeError` is raised. @@ -331,13 +331,13 @@ Other objects ``O&`` (object) [*converter*, *anything*] Convert a Python object to a C variable through a *converter* function. This takes two arguments: the first is a function, the second is the address of a C - variable (of arbitrary type), converted to :c:type:`void \*`. The *converter* + variable (of arbitrary type), converted to :c:type:`void *`. The *converter* function in turn is called as follows:: status = converter(object, address); where *object* is the Python object to be converted and *address* is the - :c:type:`void\*` argument that was passed to the :c:func:`PyArg_Parse\*` function. + :c:type:`void*` argument that was passed to the :c:func:`PyArg_Parse\*` function. The returned *status* should be ``1`` for a successful conversion and ``0`` if the conversion has failed. When the conversion fails, the *converter* function should raise an exception and leave the content of *address* unmodified. @@ -483,7 +483,7 @@ API Functions *args*; it must actually be a tuple. The length of the tuple must be at least *min* and no more than *max*; *min* and *max* may be equal. Additional arguments must be passed to the function, each of which should be a pointer to a - :c:type:`PyObject\*` variable; these will be filled in with the values from + :c:type:`PyObject*` variable; these will be filled in with the values from *args*; they will contain borrowed references. The variables which correspond to optional parameters not given by *args* will not be filled in; these should be initialized by the caller. This function returns true on success and false if @@ -652,8 +652,8 @@ Building values ``O&`` (object) [*converter*, *anything*] Convert *anything* to a Python object through a *converter* function. The - function is called with *anything* (which should be compatible with :c:type:`void - \*`) as its argument and should return a "new" Python object, or ``NULL`` if an + function is called with *anything* (which should be compatible with :c:type:`void*`) + as its argument and should return a "new" Python object, or ``NULL`` if an error occurred. ``(items)`` (:class:`tuple`) [*matching-items*] diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index 4e6fa0b32e669..dd091bb3d1003 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -89,7 +89,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. .. c:type:: Py_buffer - .. c:member:: void \*buf + .. c:member:: void *buf A pointer to the start of the logical structure described by the buffer fields. This can be any location within the underlying physical memory @@ -99,7 +99,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. For :term:`contiguous` arrays, the value points to the beginning of the memory block. - .. c:member:: void \*obj + .. c:member:: void *obj A new reference to the exporting object. The reference is owned by the consumer and automatically decremented and set to ``NULL`` by @@ -145,7 +145,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``. - .. c:member:: const char \*format + .. c:member:: const char *format A *NUL* terminated string in :mod:`struct` module style syntax describing the contents of a single item. If this is ``NULL``, ``"B"`` (unsigned bytes) @@ -164,7 +164,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. to 64. Exporters MUST respect this limit, consumers of multi-dimensional buffers SHOULD be able to handle up to :c:macro:`PyBUF_MAX_NDIM` dimensions. - .. c:member:: Py_ssize_t \*shape + .. c:member:: Py_ssize_t *shape An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` indicating the shape of the memory as an n-dimensional array. Note that @@ -177,7 +177,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. The shape array is read-only for the consumer. - .. c:member:: Py_ssize_t \*strides + .. c:member:: Py_ssize_t *strides An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` giving the number of bytes to skip to get to a new element in each @@ -189,7 +189,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. The strides array is read-only for the consumer. - .. c:member:: Py_ssize_t \*suboffsets + .. c:member:: Py_ssize_t *suboffsets An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`. If ``suboffsets[n] >= 0``, the values stored along the nth dimension are @@ -207,7 +207,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`. The suboffsets array is read-only for the consumer. - .. c:member:: void \*internal + .. c:member:: void *internal This is for use internally by the exporting object. For example, this might be re-cast as an integer by the exporter and used to store flags @@ -301,7 +301,7 @@ must be C-contiguous. +-----------------------------------+-------+---------+------------+--------+ | .. c:macro:: PyBUF_ANY_CONTIGUOUS | yes | yes | NULL | C or F | +-----------------------------------+-------+---------+------------+--------+ -| .. c:macro:: PyBUF_ND | yes | NULL | NULL | C | +| :c:macro:`PyBUF_ND` | yes | NULL | NULL | C | +-----------------------------------+-------+---------+------------+--------+ @@ -438,12 +438,12 @@ Buffer-related functions Send a request to *exporter* to fill in *view* as specified by *flags*. If the exporter cannot provide a buffer of the exact type, it MUST raise - :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to ``NULL`` and + :c:data:`PyExc_BufferError`, set ``view->obj`` to ``NULL`` and return ``-1``. - On success, fill in *view*, set :c:member:`view->obj` to a new reference + On success, fill in *view*, set ``view->obj`` to a new reference to *exporter* and return 0. In the case of chained buffer providers - that redirect requests to a single object, :c:member:`view->obj` MAY + that redirect requests to a single object, ``view->obj`` MAY refer to this object instead of *exporter* (See :ref:`Buffer Object Structures `). Successful calls to :c:func:`PyObject_GetBuffer` must be paired with calls @@ -455,7 +455,7 @@ Buffer-related functions .. c:function:: void PyBuffer_Release(Py_buffer *view) Release the buffer *view* and decrement the reference count for - :c:member:`view->obj`. This function MUST be called when the buffer + ``view->obj``. This function MUST be called when the buffer is no longer being used, otherwise reference leaks may occur. It is an error to call this function on a buffer that was not obtained via @@ -514,9 +514,9 @@ Buffer-related functions *view* as specified by flags, unless *buf* has been designated as read-only and :c:macro:`PyBUF_WRITABLE` is set in *flags*. - On success, set :c:member:`view->obj` to a new reference to *exporter* and + On success, set ``view->obj`` to a new reference to *exporter* and return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set - :c:member:`view->obj` to ``NULL`` and return ``-1``; + ``view->obj`` to ``NULL`` and return ``-1``; If this function is used as part of a :ref:`getbufferproc `, *exporter* MUST be set to the exporting object and *flags* must be passed diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst index 78e21140b2f80..5eb313c89bfd5 100644 --- a/Doc/c-api/capsule.rst +++ b/Doc/c-api/capsule.rst @@ -15,7 +15,7 @@ Refer to :ref:`using-capsules` for more information on using these objects. .. c:type:: PyCapsule This subtype of :c:type:`PyObject` represents an opaque value, useful for C - extension modules who need to pass an opaque value (as a :c:type:`void\*` + extension modules who need to pass an opaque value (as a :c:type:`void*` pointer) through Python code to other C code. It is often used to make a C function pointer defined in one module available to other modules, so the regular import mechanism can be used to access C APIs defined in dynamically diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index e48c11d336b8c..30905271c396a 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -73,7 +73,7 @@ Dictionary Objects .. index:: single: PyUnicode_FromString() Insert *val* into the dictionary *p* using *key* as a key. *key* should - be a :c:type:`const char\*`. The key object is created using + be a :c:type:`const char*`. The key object is created using ``PyUnicode_FromString(key)``. Return ``0`` on success or ``-1`` on failure. This function *does not* steal a reference to *val*. @@ -112,7 +112,7 @@ Dictionary Objects .. c:function:: PyObject* PyDict_GetItemString(PyObject *p, const char *key) This is the same as :c:func:`PyDict_GetItem`, but *key* is specified as a - :c:type:`const char\*`, rather than a :c:type:`PyObject\*`. + :c:type:`const char*`, rather than a :c:type:`PyObject*`. Note that exceptions which occur while calling :meth:`__hash__` and :meth:`__eq__` methods and creating a temporary string object @@ -161,7 +161,7 @@ Dictionary Objects prior to the first call to this function to start the iteration; the function returns true for each pair in the dictionary, and false once all pairs have been reported. The parameters *pkey* and *pvalue* should either - point to :c:type:`PyObject\*` variables that will be filled in with each key + point to :c:type:`PyObject*` variables that will be filled in with each key and value, respectively, or may be ``NULL``. Any references returned through them are borrowed. *ppos* should not be altered during iteration. Its value represents offsets within the internal dictionary structure, and diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index bbc1ee28d9b08..a4c263f47ab63 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -771,7 +771,7 @@ Standard Exceptions All standard Python exceptions are available as global variables whose names are ``PyExc_`` followed by the Python exception name. These have the type -:c:type:`PyObject\*`; they are all class objects. For completeness, here are all +:c:type:`PyObject*`; they are all class objects. For completeness, here are all the variables: .. index:: @@ -991,7 +991,7 @@ Standard Warning Categories All standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python exception name. These have the type -:c:type:`PyObject\*`; they are all class objects. For completeness, here are all +:c:type:`PyObject*`; they are all class objects. For completeness, here are all the variables: .. index:: diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst index 5370c4e350a0b..ea027ee975c65 100644 --- a/Doc/c-api/file.rst +++ b/Doc/c-api/file.rst @@ -8,7 +8,7 @@ File Objects .. index:: object: file These APIs are a minimal emulation of the Python 2 C API for built-in file -objects, which used to rely on the buffered I/O (:c:type:`FILE\*`) support +objects, which used to rely on the buffered I/O (:c:type:`FILE*`) support from the C standard library. In Python 3, files and streams use the new :mod:`io` module, which defines several layers over the low-level unbuffered I/O of the operating system. The functions described below are @@ -17,7 +17,7 @@ error reporting in the interpreter; third-party code is advised to access the :mod:`io` APIs instead. -.. c:function:: PyFile_FromFd(int fd, const char *name, const char *mode, int buffering, const char *encoding, const char *errors, const char *newline, int closefd) +.. c:function:: PyObject* PyFile_FromFd(int fd, const char *name, const char *mode, int buffering, const char *encoding, const char *errors, const char *newline, int closefd) Create a Python file object from the file descriptor of an already opened file *fd*. The arguments *name*, *encoding*, *errors* and *newline* diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 68d892dcae404..e12f7c6262c22 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -81,7 +81,7 @@ When a flag is set by an option, the value of the flag is the number of times that the option was set. For example, ``-b`` sets :c:data:`Py_BytesWarningFlag` to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. -.. c:var:: Py_BytesWarningFlag +.. c:var:: int Py_BytesWarningFlag Issue a warning when comparing :class:`bytes` or :class:`bytearray` with :class:`str` or :class:`bytes` with :class:`int`. Issue an error if greater @@ -89,7 +89,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-b` option. -.. c:var:: Py_DebugFlag +.. c:var:: int Py_DebugFlag Turn on parser debugging output (for expert only, depending on compilation options). @@ -97,7 +97,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-d` option and the :envvar:`PYTHONDEBUG` environment variable. -.. c:var:: Py_DontWriteBytecodeFlag +.. c:var:: int Py_DontWriteBytecodeFlag If set to non-zero, Python won't try to write ``.pyc`` files on the import of source modules. @@ -105,14 +105,14 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-B` option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable. -.. c:var:: Py_FrozenFlag +.. c:var:: int Py_FrozenFlag Suppress error messages when calculating the module search path in :c:func:`Py_GetPath`. Private flag used by ``_freeze_importlib`` and ``frozenmain`` programs. -.. c:var:: Py_HashRandomizationFlag +.. c:var:: int Py_HashRandomizationFlag Set to ``1`` if the :envvar:`PYTHONHASHSEED` environment variable is set to a non-empty string. @@ -120,14 +120,14 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. If the flag is non-zero, read the :envvar:`PYTHONHASHSEED` environment variable to initialize the secret hash seed. -.. c:var:: Py_IgnoreEnvironmentFlag +.. c:var:: int Py_IgnoreEnvironmentFlag Ignore all :envvar:`PYTHON*` environment variables, e.g. :envvar:`PYTHONPATH` and :envvar:`PYTHONHOME`, that might be set. Set by the :option:`-E` and :option:`-I` options. -.. c:var:: Py_InspectFlag +.. c:var:: int Py_InspectFlag When a script is passed as first argument or the :option:`-c` option is used, enter interactive mode after executing the script or the command, even when @@ -136,11 +136,11 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-i` option and the :envvar:`PYTHONINSPECT` environment variable. -.. c:var:: Py_InteractiveFlag +.. c:var:: int Py_InteractiveFlag Set by the :option:`-i` option. -.. c:var:: Py_IsolatedFlag +.. c:var:: int Py_IsolatedFlag Run Python in isolated mode. In isolated mode :data:`sys.path` contains neither the script's directory nor the user's site-packages directory. @@ -149,7 +149,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. versionadded:: 3.4 -.. c:var:: Py_LegacyWindowsFSEncodingFlag +.. c:var:: int Py_LegacyWindowsFSEncodingFlag If the flag is non-zero, use the ``mbcs`` encoding instead of the UTF-8 encoding for the filesystem encoding. @@ -161,7 +161,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. availability:: Windows. -.. c:var:: Py_LegacyWindowsStdioFlag +.. c:var:: int Py_LegacyWindowsStdioFlag If the flag is non-zero, use :class:`io.FileIO` instead of :class:`WindowsConsoleIO` for :mod:`sys` standard streams. @@ -173,7 +173,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. availability:: Windows. -.. c:var:: Py_NoSiteFlag +.. c:var:: int Py_NoSiteFlag Disable the import of the module :mod:`site` and the site-dependent manipulations of :data:`sys.path` that it entails. Also disable these @@ -182,7 +182,7 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-S` option. -.. c:var:: Py_NoUserSiteDirectory +.. c:var:: int Py_NoUserSiteDirectory Don't add the :data:`user site-packages directory ` to :data:`sys.path`. @@ -190,12 +190,12 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. Set by the :option:`-s` and :option:`-I` options, and the :envvar:`PYTHONNOUSERSITE` environment variable. -.. c:var:: Py_OptimizeFlag +.. c:var:: int Py_OptimizeFlag Set by the :option:`-O` option and the :envvar:`PYTHONOPTIMIZE` environment variable. -.. c:var:: Py_QuietFlag +.. c:var:: int Py_QuietFlag Don't display the copyright and version messages even in interactive mode. @@ -203,14 +203,14 @@ to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2. .. versionadded:: 3.2 -.. c:var:: Py_UnbufferedStdioFlag +.. c:var:: int Py_UnbufferedStdioFlag Force the stdout and stderr streams to be unbuffered. Set by the :option:`-u` option and the :envvar:`PYTHONUNBUFFERED` environment variable. -.. c:var:: Py_VerboseFlag +.. c:var:: int Py_VerboseFlag Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded. If greater or equal @@ -830,7 +830,7 @@ code, or when embedding the Python interpreter: .. c:type:: PyThreadState This data structure represents the state of a single thread. The only public - data member is :c:type:`PyInterpreterState \*`:attr:`interp`, which points to + data member is :attr:`interp` (:c:type:`PyInterpreterState *`), which points to this thread's interpreter state. @@ -1516,7 +1516,7 @@ The Python interpreter provides low-level support for thread-local storage (TLS) which wraps the underlying native TLS implementation to support the Python-level thread local storage API (:class:`threading.local`). The CPython C level APIs are similar to those offered by pthreads and Windows: -use a thread key and functions to associate a :c:type:`void\*` value per +use a thread key and functions to associate a :c:type:`void*` value per thread. The GIL does *not* need to be held when calling these functions; they supply @@ -1527,8 +1527,8 @@ you need to include :file:`pythread.h` to use thread-local storage. .. note:: None of these API functions handle memory management on behalf of the - :c:type:`void\*` values. You need to allocate and deallocate them yourself. - If the :c:type:`void\*` values happen to be :c:type:`PyObject\*`, these + :c:type:`void*` values. You need to allocate and deallocate them yourself. + If the :c:type:`void*` values happen to be :c:type:`PyObject*`, these functions don't do refcount operations on them either. .. _thread-specific-storage-api: @@ -1624,14 +1624,14 @@ undefined if the given :c:type:`Py_tss_t` has not been initialized by .. c:function:: int PyThread_tss_set(Py_tss_t *key, void *value) - Return a zero value to indicate successfully associating a :c:type:`void\*` + Return a zero value to indicate successfully associating a :c:type:`void*` value with a TSS key in the current thread. Each thread has a distinct - mapping of the key to a :c:type:`void\*` value. + mapping of the key to a :c:type:`void*` value. .. c:function:: void* PyThread_tss_get(Py_tss_t *key) - Return the :c:type:`void\*` value associated with a TSS key in the current + Return the :c:type:`void*` value associated with a TSS key in the current thread. This returns ``NULL`` if no value is associated with the key in the current thread. diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index 718f40eb6c247..fd1268e310260 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -216,13 +216,13 @@ Objects, Types and Reference Counts .. index:: object: type Most Python/C API functions have one or more arguments as well as a return value -of type :c:type:`PyObject\*`. This type is a pointer to an opaque data type +of type :c:type:`PyObject*`. This type is a pointer to an opaque data type representing an arbitrary Python object. Since all Python object types are treated the same way by the Python language in most situations (e.g., assignments, scope rules, and argument passing), it is only fitting that they should be represented by a single C type. Almost all Python objects live on the heap: you never declare an automatic or static variable of type -:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject\*` can be +:c:type:`PyObject`, only pointer variables of type :c:type:`PyObject*` can be declared. The sole exception are the type objects; since these must never be deallocated, they are typically static :c:type:`PyTypeObject` objects. @@ -483,7 +483,7 @@ Types There are few other data types that play a significant role in the Python/C API; most are simple C types such as :c:type:`int`, :c:type:`long`, -:c:type:`double` and :c:type:`char\*`. A few structure types are used to +:c:type:`double` and :c:type:`char*`. A few structure types are used to describe static tables used to list the functions exported by a module or the data attributes of a new object type, and another is used to describe the value of a complex number. These will be discussed together with the functions that diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst index 7b179e22e290e..7bb0dad2b6b6d 100644 --- a/Doc/c-api/marshal.rst +++ b/Doc/c-api/marshal.rst @@ -43,7 +43,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: long PyMarshal_ReadLongFromFile(FILE *file) - Return a C :c:type:`long` from the data stream in a :c:type:`FILE\*` opened + Return a C :c:type:`long` from the data stream in a :c:type:`FILE*` opened for reading. Only a 32-bit value can be read in using this function, regardless of the native size of :c:type:`long`. @@ -53,7 +53,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: int PyMarshal_ReadShortFromFile(FILE *file) - Return a C :c:type:`short` from the data stream in a :c:type:`FILE\*` opened + Return a C :c:type:`short` from the data stream in a :c:type:`FILE*` opened for reading. Only a 16-bit value can be read in using this function, regardless of the native size of :c:type:`short`. @@ -63,7 +63,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: PyObject* PyMarshal_ReadObjectFromFile(FILE *file) - Return a Python object from the data stream in a :c:type:`FILE\*` opened for + Return a Python object from the data stream in a :c:type:`FILE*` opened for reading. On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError` @@ -72,7 +72,7 @@ The following functions allow marshalled values to be read back in. .. c:function:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file) - Return a Python object from the data stream in a :c:type:`FILE\*` opened for + Return a Python object from the data stream in a :c:type:`FILE*` opened for reading. Unlike :c:func:`PyMarshal_ReadObjectFromFile`, this function assumes that no further objects will be read from the file, allowing it to aggressively load file data into memory so that the de-serialization can diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 8a8542f0479ec..87425bcf1e71f 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -109,7 +109,7 @@ zero bytes. .. c:function:: void* PyMem_RawMalloc(size_t n) - Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as @@ -120,7 +120,7 @@ zero bytes. .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) Allocates *nelem* elements each whose size in bytes is *elsize* and returns - a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the + a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. The memory is initialized to zeros. Requesting zero elements or elements of size zero bytes returns a distinct @@ -180,7 +180,7 @@ The :ref:`default memory allocator ` uses the .. c:function:: void* PyMem_Malloc(size_t n) - Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as @@ -191,7 +191,7 @@ The :ref:`default memory allocator ` uses the .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) Allocates *nelem* elements each whose size in bytes is *elsize* and returns - a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the + a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. The memory is initialized to zeros. Requesting zero elements or elements of size zero bytes returns a distinct @@ -233,14 +233,14 @@ The following type-oriented macros are provided for convenience. Note that .. c:function:: TYPE* PyMem_New(TYPE, size_t n) Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of - memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have + memory. Returns a pointer cast to :c:type:`TYPE*`. The memory will not have been initialized in any way. .. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * - sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, + sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE*`. On return, *p* will be a pointer to the new memory area, or ``NULL`` in the event of failure. @@ -282,7 +282,7 @@ The :ref:`default object allocator ` uses the .. c:function:: void* PyObject_Malloc(size_t n) - Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + Allocates *n* bytes and returns a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as @@ -293,7 +293,7 @@ The :ref:`default object allocator ` uses the .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize) Allocates *nelem* elements each whose size in bytes is *elsize* and returns - a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the + a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the request fails. The memory is initialized to zeros. Requesting zero elements or elements of size zero bytes returns a distinct @@ -388,7 +388,7 @@ Customize Memory Allocators Enum used to identify an allocator domain. Domains: - .. c:var:: PYMEM_DOMAIN_RAW + .. c:macro:: PYMEM_DOMAIN_RAW Functions: @@ -397,7 +397,7 @@ Customize Memory Allocators * :c:func:`PyMem_RawCalloc` * :c:func:`PyMem_RawFree` - .. c:var:: PYMEM_DOMAIN_MEM + .. c:macro:: PYMEM_DOMAIN_MEM Functions: @@ -406,7 +406,7 @@ Customize Memory Allocators * :c:func:`PyMem_Calloc` * :c:func:`PyMem_Free` - .. c:var:: PYMEM_DOMAIN_OBJ + .. c:macro:: PYMEM_DOMAIN_OBJ Functions: @@ -519,11 +519,11 @@ Customize pymalloc Arena Allocator | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena | +--------------------------------------------------+---------------------------------------+ -.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) +.. c:function:: void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) Get the arena allocator. -.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) +.. c:function:: void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) Set the arena allocator. diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index d2b8f4c12503e..f840dd90558cc 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -301,7 +301,7 @@ The *m_slots* array must be terminated by a slot with id 0. The available slot types are: -.. c:var:: Py_mod_create +.. c:macro:: Py_mod_create Specifies a function that is called to create the module object itself. The *value* pointer of this slot must point to a function of the signature: @@ -333,7 +333,7 @@ The available slot types are: ``PyModuleDef`` has non-``NULL`` ``m_traverse``, ``m_clear``, ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``. -.. c:var:: Py_mod_exec +.. c:macro:: Py_mod_exec Specifies a function that is called to *execute* the module. This is equivalent to executing the code of a Python module: typically, diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 3bdc0f56eb251..d4e8b7468089c 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -290,7 +290,7 @@ Object Protocol This is the equivalent of the Python expression: ``callable(*args)``. - Note that if you only pass :c:type:`PyObject \*` args, + Note that if you only pass :c:type:`PyObject *` args, :c:func:`PyObject_CallFunctionObjArgs` is a faster alternative. .. versionchanged:: 3.4 @@ -311,17 +311,17 @@ Object Protocol This is the equivalent of the Python expression: ``obj.name(arg1, arg2, ...)``. - Note that if you only pass :c:type:`PyObject \*` args, + Note that if you only pass :c:type:`PyObject *` args, :c:func:`PyObject_CallMethodObjArgs` is a faster alternative. .. versionchanged:: 3.4 The types of *name* and *format* were changed from ``char *``. -.. c:function:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ..., NULL) +.. c:function:: PyObject* PyObject_CallFunctionObjArgs(PyObject *callable, ...) Call a callable Python object *callable*, with a variable number of - :c:type:`PyObject\*` arguments. The arguments are provided as a variable number + :c:type:`PyObject*` arguments. The arguments are provided as a variable number of parameters followed by ``NULL``. Return the result of the call on success, or raise an exception and return @@ -331,11 +331,11 @@ Object Protocol ``callable(arg1, arg2, ...)``. -.. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ..., NULL) +.. c:function:: PyObject* PyObject_CallMethodObjArgs(PyObject *obj, PyObject *name, ...) Calls a method of the Python object *obj*, where the name of the method is given as a Python string object in *name*. It is called with a variable number of - :c:type:`PyObject\*` arguments. The arguments are provided as a variable number + :c:type:`PyObject*` arguments. The arguments are provided as a variable number of parameters followed by ``NULL``. Return the result of the call on success, or raise an exception and return @@ -377,7 +377,7 @@ Object Protocol .. versionadded:: 3.8 -.. c:var:: PY_VECTORCALL_ARGUMENTS_OFFSET +.. c:macro:: PY_VECTORCALL_ARGUMENTS_OFFSET If set in a vectorcall *nargsf* argument, the callee is allowed to temporarily change ``args[-1]``. In other words, *args* points to @@ -463,7 +463,7 @@ Object Protocol is equivalent to the Python expression ``type(o)``. This function increments the reference count of the return value. There's really no reason to use this function instead of the common expression ``o->ob_type``, which returns a - pointer of type :c:type:`PyTypeObject\*`, except when the incremented reference + pointer of type :c:type:`PyTypeObject*`, except when the incremented reference count is needed. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 2dc0762ee7fb8..e9c276ce78290 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -105,7 +105,7 @@ the definition of all other Python objects. .. c:type:: PyCFunction Type of the functions used to implement most Python callables in C. - Functions of this type take two :c:type:`PyObject\*` parameters and return + Functions of this type take two :c:type:`PyObject*` parameters and return one such value. If the return value is ``NULL``, an exception shall have been set. If not ``NULL``, the return value is interpreted as the return value of the function as exposed in Python. The function must return a new @@ -151,10 +151,10 @@ the definition of all other Python objects. +------------------+---------------+-------------------------------+ The :attr:`ml_meth` is a C function pointer. The functions may be of different -types, but they always return :c:type:`PyObject\*`. If the function is not of +types, but they always return :c:type:`PyObject*`. If the function is not of the :c:type:`PyCFunction`, the compiler will require a cast in the method table. Even though :c:type:`PyCFunction` defines the first parameter as -:c:type:`PyObject\*`, it is common that the method implementation uses the +:c:type:`PyObject*`, it is common that the method implementation uses the specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. @@ -168,7 +168,7 @@ also keyword arguments. So there are a total of 6 calling conventions: .. data:: METH_VARARGS This is the typical calling convention, where the methods have the type - :c:type:`PyCFunction`. The function expects two :c:type:`PyObject\*` values. + :c:type:`PyCFunction`. The function expects two :c:type:`PyObject*` values. The first one is the *self* object for methods; for module functions, it is the module object. The second parameter (often called *args*) is a tuple object representing all arguments. This parameter is typically processed @@ -189,7 +189,7 @@ also keyword arguments. So there are a total of 6 calling conventions: Fast calling convention supporting only positional arguments. The methods have the type :c:type:`_PyCFunctionFast`. The first parameter is *self*, the second parameter is a C array - of :c:type:`PyObject\*` values indicating the arguments and the third + of :c:type:`PyObject*` values indicating the arguments and the third parameter is the number of arguments (the length of the array). This is not part of the :ref:`limited API `. @@ -202,7 +202,7 @@ also keyword arguments. So there are a total of 6 calling conventions: Extension of :const:`METH_FASTCALL` supporting also keyword arguments, with methods of type :c:type:`_PyCFunctionFastWithKeywords`. Keyword arguments are passed the same way as in the vectorcall protocol: - there is an additional fourth :c:type:`PyObject\*` parameter + there is an additional fourth :c:type:`PyObject*` parameter which is a tuple representing the names of the keyword arguments or possibly ``NULL`` if there are no keywords. The values of the keyword arguments are stored in the *args* array, after the positional arguments. @@ -226,7 +226,7 @@ also keyword arguments. So there are a total of 6 calling conventions: Methods with a single object argument can be listed with the :const:`METH_O` flag, instead of invoking :c:func:`PyArg_ParseTuple` with a ``"O"`` argument. They have the type :c:type:`PyCFunction`, with the *self* parameter, and a - :c:type:`PyObject\*` parameter representing the single argument. + :c:type:`PyObject*` parameter representing the single argument. These two constants are not used to indicate the calling convention but the @@ -359,7 +359,7 @@ definition with the same method name. | | | getter and setter | +-------------+------------------+-----------------------------------+ - The ``get`` function takes one :c:type:`PyObject\*` parameter (the + The ``get`` function takes one :c:type:`PyObject*` parameter (the instance) and a function pointer (the associated ``closure``):: typedef PyObject *(*getter)(PyObject *, void *); @@ -367,7 +367,7 @@ definition with the same method name. It should return a new reference on success or ``NULL`` with a set exception on failure. - ``set`` functions take two :c:type:`PyObject\*` parameters (the instance and + ``set`` functions take two :c:type:`PyObject*` parameters (the instance and the value to be set) and a function pointer (the associated ``closure``):: typedef int (*setter)(PyObject *, PyObject *, void *); diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst index 25df3974e870e..d7acc4eaa979a 100644 --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -166,7 +166,7 @@ type. .. c:type:: PyStructSequence_Field Describes a field of a struct sequence. As a struct sequence is modeled as a - tuple, all fields are typed as :c:type:`PyObject\*`. The index in the + tuple, all fields are typed as :c:type:`PyObject*`. The index in the :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which field of the struct sequence is described. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 163f599d1c264..3d186231b26b7 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -1360,7 +1360,7 @@ and :c:type:`PyType_Type` effectively act as defaults.) The following macro is defined to ease writing rich comparison functions: - .. c:function:: PyObject \*Py_RETURN_RICHCOMPARE(VAL_A, VAL_B, int op) + .. c:macro:: Py_RETURN_RICHCOMPARE(VAL_A, VAL_B, op) Return ``Py_True`` or ``Py_False`` from the function, depending on the result of a comparison. @@ -1398,7 +1398,7 @@ and :c:type:`PyType_Type` effectively act as defaults.) than zero and contains the offset in the instance structure of the weak reference list head (ignoring the GC header, if present); this offset is used by :c:func:`PyObject_ClearWeakRefs` and the :c:func:`PyWeakref_\*` functions. The - instance structure needs to include a field of type :c:type:`PyObject\*` which is + instance structure needs to include a field of type :c:type:`PyObject*` which is initialized to ``NULL``. Do not confuse this field with :c:member:`~PyTypeObject.tp_weaklist`; that is the list head for diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index a70b1d0afd2d5..9b68c31e541ae 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -199,7 +199,7 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 -.. c:function:: PyUnicode_MAX_CHAR_VALUE(PyObject *o) +.. c:macro:: PyUnicode_MAX_CHAR_VALUE(o) Return the maximum code point that is suitable for creating another string based on *o*, which must be in the "canonical" representation. This is @@ -1498,17 +1498,21 @@ These are the mapping codec APIs: The following codec API is special in that maps Unicode to Unicode. -.. c:function:: PyObject* PyUnicode_Translate(PyObject *unicode, \ - PyObject *mapping, const char *errors) +.. c:function:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, const char *errors) - Translate a Unicode object using the given *mapping* object and return the - resulting Unicode object. Return ``NULL`` if an exception was raised by the + Translate a string by applying a character mapping table to it and return the + resulting Unicode object. Return ``NULL`` if an exception was raised by the codec. - The *mapping* object must map Unicode ordinal integers to Unicode strings, - integers (which are then interpreted as Unicode ordinals) or ``None`` - (causing deletion of the character). Unmapped character ordinals (ones - which cause a :exc:`LookupError`) are left untouched and are copied as-is. + The mapping table must map Unicode ordinal integers to Unicode ordinal integers + or ``None`` (causing deletion of the character). + + Mapping tables need only provide the :meth:`__getitem__` interface; dictionaries + and sequences work well. Unmapped character ordinals (ones which cause a + :exc:`LookupError`) are left untouched and are copied as-is. + + *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to + use the default error handling. .. c:function:: PyObject* PyUnicode_TranslateCharmap(const Py_UNICODE *s, Py_ssize_t size, \ @@ -1611,23 +1615,6 @@ They all return ``NULL`` or ``-1`` if an exception occurs. characters are not included in the resulting strings. -.. c:function:: PyObject* PyUnicode_Translate(PyObject *str, PyObject *table, \ - const char *errors) - - Translate a string by applying a character mapping table to it and return the - resulting Unicode object. - - The mapping table must map Unicode ordinal integers to Unicode ordinal integers - or ``None`` (causing deletion of the character). - - Mapping tables need only provide the :meth:`__getitem__` interface; dictionaries - and sequences work well. Unmapped character ordinals (ones which cause a - :exc:`LookupError`) are left untouched and are copied as-is. - - *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to - use the default error handling. - - .. c:function:: PyObject* PyUnicode_Join(PyObject *separator, PyObject *seq) Join a sequence of strings using the given *separator* and return the resulting diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 98cea52ab7193..208a14e62a0e3 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -16,11 +16,11 @@ parameter. The available start symbols are :const:`Py_eval_input`, :const:`Py_file_input`, and :const:`Py_single_input`. These are described following the functions which accept them as parameters. -Note also that several of these functions take :c:type:`FILE\*` parameters. One +Note also that several of these functions take :c:type:`FILE*` parameters. One particular issue which needs to be handled carefully is that the :c:type:`FILE` structure for different C libraries can be different and incompatible. Under Windows (at least), it is possible for dynamically linked extensions to actually -use different libraries, so care should be taken that :c:type:`FILE\*` parameters +use different libraries, so care should be taken that :c:type:`FILE*` parameters are only passed to these functions if it is certain that they were created by the same library that the Python runtime is using. diff --git a/Doc/conf.py b/Doc/conf.py index abaa760c98c1a..4cca13b119374 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -226,3 +226,13 @@ # Relative filename of the reference count data file. refcount_file = 'data/refcounts.dat' + +# Sphinx 2 and Sphinx 3 compatibility +# ----------------------------------- + +# bpo-40204: Allow Sphinx 2 syntax in the C domain +c_allow_pre_v3 = True + +# bpo-40204: Disable warnings on Sphinx 2 syntax of the C domain since the +# documentation is built with -W (warnings treated as errors). +c_warn_on_allowed_pre_v3 = False diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 0eb6ffd026f49..4da77e797d222 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -416,7 +416,7 @@ But this would be risky. Our type doesn't restrict the type of the ``first`` member, so it could be any kind of object. It could have a destructor that causes code to be executed that tries to access the ``first`` member; or that destructor could release the -:term:`Global interpreter Lock` and let arbitrary code run in other +:term:`Global interpreter Lock ` and let arbitrary code run in other threads that accesses and modifies our object. To be paranoid and protect ourselves against this possibility, we almost diff --git a/Doc/glossary.rst b/Doc/glossary.rst index e997d366777b3..7be755e411310 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -587,7 +587,7 @@ Glossary and :class:`tuple`) and some non-sequence types like :class:`dict`, :term:`file objects `, and objects of any classes you define with an :meth:`__iter__` method or with a :meth:`__getitem__` method - that implements :term:`Sequence` semantics. + that implements :term:`Sequence ` semantics. Iterables can be used in a :keyword:`for` loop and in many other places where a sequence is diff --git a/Doc/howto/instrumentation.rst b/Doc/howto/instrumentation.rst index 909deb5fed33f..f0081e4ec2890 100644 --- a/Doc/howto/instrumentation.rst +++ b/Doc/howto/instrumentation.rst @@ -272,9 +272,7 @@ should instead read: Available static markers ------------------------ -.. I'm reusing the "c:function" type for markers - -.. c:function:: function__entry(str filename, str funcname, int lineno) +.. object:: function__entry(str filename, str funcname, int lineno) This marker indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions. @@ -290,7 +288,7 @@ Available static markers * ``$arg3`` : ``int`` line number -.. c:function:: function__return(str filename, str funcname, int lineno) +.. object:: function__return(str filename, str funcname, int lineno) This marker is the converse of :c:func:`function__entry`, and indicates that execution of a Python function has ended (either via ``return``, or via an @@ -298,7 +296,7 @@ Available static markers The arguments are the same as for :c:func:`function__entry` -.. c:function:: line(str filename, str funcname, int lineno) +.. object:: line(str filename, str funcname, int lineno) This marker indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. It is @@ -306,24 +304,24 @@ Available static markers The arguments are the same as for :c:func:`function__entry`. -.. c:function:: gc__start(int generation) +.. object:: gc__start(int generation) Fires when the Python interpreter starts a garbage collection cycle. ``arg0`` is the generation to scan, like :func:`gc.collect()`. -.. c:function:: gc__done(long collected) +.. object:: gc__done(long collected) Fires when the Python interpreter finishes a garbage collection cycle. ``arg0`` is the number of collected objects. -.. c:function:: import__find__load__start(str modulename) +.. object:: import__find__load__start(str modulename) Fires before :mod:`importlib` attempts to find and load the module. ``arg0`` is the module name. .. versionadded:: 3.7 -.. c:function:: import__find__load__done(str modulename, int found) +.. object:: import__find__load__done(str modulename, int found) Fires after :mod:`importlib`'s find_and_load function is called. ``arg0`` is the module name, ``arg1`` indicates if module was @@ -332,7 +330,7 @@ Available static markers .. versionadded:: 3.7 -.. c:function:: audit(str event, void *tuple) +.. object:: audit(str event, void *tuple) Fires when :func:`sys.audit` or :c:func:`PySys_Audit` is called. ``arg0`` is the event name as C string, ``arg1`` is a :c:type:`PyObject` @@ -375,14 +373,14 @@ If this file is installed in SystemTap's tapset directory (e.g. ``/usr/share/systemtap/tapset``), then these additional probepoints become available: -.. c:function:: python.function.entry(str filename, str funcname, int lineno, frameptr) +.. object:: python.function.entry(str filename, str funcname, int lineno, frameptr) This probe point indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions. -.. c:function:: python.function.return(str filename, str funcname, int lineno, frameptr) +.. object:: python.function.return(str filename, str funcname, int lineno, frameptr) - This probe point is the converse of :c:func:`python.function.return`, and + This probe point is the converse of ``python.function.return``, and indicates that execution of a Python function has ended (either via ``return``, or via an exception). It is only triggered for pure-Python (bytecode) functions. diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst index 7328907730fb1..2e917cf7321b8 100644 --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -208,6 +208,7 @@ number of frames must be filled in. .. method:: aifc.tell() + :noindex: Return the current write position in the output file. Useful in combination with :meth:`setmark`. @@ -232,6 +233,7 @@ number of frames must be filled in. .. method:: aifc.close() + :noindex: Close the AIFF file. The header of the file is updated to reflect the actual size of the audio data. After calling this method, the object can no longer be diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 2a3fb142f7297..dc7ae30b6d2fa 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -185,7 +185,7 @@ ABC Inherits from Abstract Methods Mixin expressions. Custom implementations must provide the :meth:`__await__` method. - :term:`Coroutine` objects and instances of the + :term:`Coroutine ` objects and instances of the :class:`~collections.abc.Coroutine` ABC are all instances of this ABC. .. note:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 41d47c7ef1530..88c1d23a55256 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -209,7 +209,8 @@ ProcessPoolExecutor The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that uses a pool of processes to execute calls asynchronously. :class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which -allows it to side-step the :term:`Global Interpreter Lock` but also means that +allows it to side-step the :term:`Global Interpreter Lock +` but also means that only picklable objects can be executed and returned. The ``__main__`` module must be importable by worker subprocesses. This means diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 739477f55fddd..2e22a549ee281 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -674,97 +674,98 @@ be overridden by subclasses or by attribute assignment. .. attribute:: ConfigParser.BOOLEAN_STATES - By default when using :meth:`~ConfigParser.getboolean`, config parsers - consider the following values ``True``: ``'1'``, ``'yes'``, ``'true'``, - ``'on'`` and the following values ``False``: ``'0'``, ``'no'``, ``'false'``, - ``'off'``. You can override this by specifying a custom dictionary of strings - and their Boolean outcomes. For example: - - .. doctest:: - - >>> custom = configparser.ConfigParser() - >>> custom['section1'] = {'funky': 'nope'} - >>> custom['section1'].getboolean('funky') - Traceback (most recent call last): - ... - ValueError: Not a boolean: nope - >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} - >>> custom['section1'].getboolean('funky') - False - - Other typical Boolean pairs include ``accept``/``reject`` or - ``enabled``/``disabled``. + By default when using :meth:`~ConfigParser.getboolean`, config parsers + consider the following values ``True``: ``'1'``, ``'yes'``, ``'true'``, + ``'on'`` and the following values ``False``: ``'0'``, ``'no'``, ``'false'``, + ``'off'``. You can override this by specifying a custom dictionary of strings + and their Boolean outcomes. For example: + + .. doctest:: + + >>> custom = configparser.ConfigParser() + >>> custom['section1'] = {'funky': 'nope'} + >>> custom['section1'].getboolean('funky') + Traceback (most recent call last): + ... + ValueError: Not a boolean: nope + >>> custom.BOOLEAN_STATES = {'sure': True, 'nope': False} + >>> custom['section1'].getboolean('funky') + False + + Other typical Boolean pairs include ``accept``/``reject`` or + ``enabled``/``disabled``. .. method:: ConfigParser.optionxform(option) + :noindex: - This method transforms option names on every read, get, or set - operation. The default converts the name to lowercase. This also - means that when a configuration file gets written, all keys will be - lowercase. Override this method if that's unsuitable. - For example: + This method transforms option names on every read, get, or set + operation. The default converts the name to lowercase. This also + means that when a configuration file gets written, all keys will be + lowercase. Override this method if that's unsuitable. + For example: - .. doctest:: + .. doctest:: + + >>> config = """ + ... [Section1] + ... Key = Value + ... + ... [Section2] + ... AnotherKey = Value + ... """ + >>> typical = configparser.ConfigParser() + >>> typical.read_string(config) + >>> list(typical['Section1'].keys()) + ['key'] + >>> list(typical['Section2'].keys()) + ['anotherkey'] + >>> custom = configparser.RawConfigParser() + >>> custom.optionxform = lambda option: option + >>> custom.read_string(config) + >>> list(custom['Section1'].keys()) + ['Key'] + >>> list(custom['Section2'].keys()) + ['AnotherKey'] - >>> config = """ - ... [Section1] - ... Key = Value - ... - ... [Section2] - ... AnotherKey = Value - ... """ - >>> typical = configparser.ConfigParser() - >>> typical.read_string(config) - >>> list(typical['Section1'].keys()) - ['key'] - >>> list(typical['Section2'].keys()) - ['anotherkey'] - >>> custom = configparser.RawConfigParser() - >>> custom.optionxform = lambda option: option - >>> custom.read_string(config) - >>> list(custom['Section1'].keys()) - ['Key'] - >>> list(custom['Section2'].keys()) - ['AnotherKey'] - - .. note:: - The optionxform function transforms option names to a canonical form. - This should be an idempotent function: if the name is already in - canonical form, it should be returned unchanged. + .. note:: + The optionxform function transforms option names to a canonical form. + This should be an idempotent function: if the name is already in + canonical form, it should be returned unchanged. .. attribute:: ConfigParser.SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered - part of the section name, thus ``[ larch ]`` will be read as a section of - name ``" larch "``. Override this attribute if that's unsuitable. For - example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: + + .. doctest:: + + >>> import re + >>> config = """ + ... [Section 1] + ... option = value + ... + ... [ Section 2 ] + ... another = val + ... """ + >>> typical = configparser.ConfigParser() + >>> typical.read_string(config) + >>> typical.sections() + ['Section 1', ' Section 2 '] + >>> custom = configparser.ConfigParser() + >>> custom.SECTCRE = re.compile(r"\[ *(?P
[^]]+?) *\]") + >>> custom.read_string(config) + >>> custom.sections() + ['Section 1', 'Section 2'] - .. doctest:: + .. note:: - >>> import re - >>> config = """ - ... [Section 1] - ... option = value - ... - ... [ Section 2 ] - ... another = val - ... """ - >>> typical = configparser.ConfigParser() - >>> typical.read_string(config) - >>> typical.sections() - ['Section 1', ' Section 2 '] - >>> custom = configparser.ConfigParser() - >>> custom.SECTCRE = re.compile(r"\[ *(?P
[^]]+?) *\]") - >>> custom.read_string(config) - >>> custom.sections() - ['Section 1', 'Section 2'] - - .. note:: - - While ConfigParser objects also use an ``OPTCRE`` attribute for recognizing - option lines, it's not recommended to override it because that would - interfere with constructor options *allow_no_value* and *delimiters*. + While ConfigParser objects also use an ``OPTCRE`` attribute for recognizing + option lines, it's not recommended to override it because that would + interfere with constructor options *allow_no_value* and *delimiters*. Legacy API Examples diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index ada311bc3a205..0837809df5902 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -24,6 +24,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. .. class:: SequenceMatcher + :noindex: This is a flexible class for comparing pairs of sequences of any type, so long as the sequence elements are :term:`hashable`. The basic algorithm predates, and is a @@ -648,6 +649,7 @@ The :class:`Differ` class has this constructor: .. class:: Differ(linejunk=None, charjunk=None) + :noindex: Optional keyword parameters *linejunk* and *charjunk* are for filter functions (or ``None``): diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index 2649b7d8a51fd..ff1510cdd0e41 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -50,6 +50,7 @@ helper, :class:`auto`. the bitwise operations without losing their :class:`Flag` membership. .. function:: unique + :noindex: Enum class decorator that ensures only one name is bound to any one value. diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 2de10435904aa..f205f5fee0544 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -1036,7 +1036,7 @@ find and load modules. .. class:: WindowsRegistryFinder - :term:`Finder` for modules declared in the Windows registry. This class + :term:`Finder ` for modules declared in the Windows registry. This class implements the :class:`importlib.abc.MetaPathFinder` ABC. Only class methods are defined by this class to alleviate the need for @@ -1051,7 +1051,7 @@ find and load modules. .. class:: PathFinder - A :term:`Finder` for :data:`sys.path` and package ``__path__`` attributes. + A :term:`Finder ` for :data:`sys.path` and package ``__path__`` attributes. This class implements the :class:`importlib.abc.MetaPathFinder` ABC. Only class methods are defined by this class to alleviate the need for diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index bb3ee87a4403a..a0f814ed140a5 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -14,7 +14,8 @@ Introduction :mod:`multiprocessing` is a package that supports spawning processes using an API similar to the :mod:`threading` module. The :mod:`multiprocessing` package offers both local and remote concurrency, effectively side-stepping the -:term:`Global Interpreter Lock` by using subprocesses instead of threads. Due +:term:`Global Interpreter Lock ` by using +subprocesses instead of threads. Due to this, the :mod:`multiprocessing` module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows. diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 78a5157345858..9a24e7a70c01e 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -68,7 +68,7 @@ support. .. class:: ImpLoader(fullname, file, filename, etc) - :term:`Loader` that wraps Python's "classic" import algorithm. + :term:`Loader ` that wraps Python's "classic" import algorithm. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index c72f22b5f6699..9ab050e8ab82c 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1627,7 +1627,9 @@ to sockets. .. method:: socket.setsockopt(level, optname, value: int) .. method:: socket.setsockopt(level, optname, value: buffer) + :noindex: .. method:: socket.setsockopt(level, optname, None, optlen: int) + :noindex: .. index:: module: struct diff --git a/Doc/library/string.rst b/Doc/library/string.rst index fa906f799c108..62e86d6dd9706 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -308,7 +308,7 @@ non-empty format specification typically modifies the result. The general form of a *standard format specifier* is: -.. productionlist:: sf +.. productionlist:: format_spec: [[`fill`]`align`][`sign`][#][0][`width`][`grouping_option`][.`precision`][`type`] fill: align: "<" | ">" | "=" | "^" diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index d60f1c8a5f2d8..87c36aa4a2cbc 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -151,6 +151,7 @@ Some facts and figures: .. class:: TarFile + :noindex: Class for reading and writing tar archives. Do not use this class directly: use :func:`tarfile.open` instead. See :ref:`tarfile-objects`. diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index f4b58d3d3aebb..4c78e6e12056a 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -395,7 +395,8 @@ since it is impossible to detect the termination of alien threads. .. impl-detail:: - In CPython, due to the :term:`Global Interpreter Lock`, only one thread + In CPython, due to the :term:`Global Interpreter Lock + `, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational diff --git a/Doc/library/token.rst b/Doc/library/token.rst index dab8f0fa9b64f..7f598cd38d7f8 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -70,6 +70,7 @@ the :mod:`tokenize` module. .. data:: TYPE_COMMENT + :noindex: Token value indicating that a type comment was recognized. Such tokens are only produced when :func:`ast.parse()` is invoked with diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index fed85045435b1..d3487537df99a 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1069,6 +1069,7 @@ More drawing control ~~~~~~~~~~~~~~~~~~~~ .. function:: reset() + :noindex: Delete the turtle's drawings from the screen, re-center the turtle and set variables to the default values. @@ -1090,6 +1091,7 @@ More drawing control .. function:: clear() + :noindex: Delete the turtle's drawings from the screen. Do not move turtle. State and position of the turtle as well as drawings of other turtles are not affected. @@ -1362,6 +1364,7 @@ Using events ------------ .. function:: onclick(fun, btn=1, add=None) + :noindex: :param fun: a function with two arguments which will be called with the coordinates of the clicked point on the canvas diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index adc0593e8094a..0edf116fe754f 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -954,7 +954,7 @@ tracking URIs for which authentication credentials should always be sent. If *is_authenticated* is specified as ``True``, *realm* is ignored. -.. method:: HTTPPasswordMgr.find_user_password(realm, authuri) +.. method:: HTTPPasswordMgrWithPriorAuth.find_user_password(realm, authuri) Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index d364e61b8cf46..c1262cdb9c106 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2596,7 +2596,7 @@ Awaitable Objects ----------------- An :term:`awaitable` object generally implements an :meth:`__await__` method. -:term:`Coroutine` objects returned from :keyword:`async def` functions +:term:`Coroutine objects ` returned from :keyword:`async def` functions are awaitable. .. note:: @@ -2621,7 +2621,7 @@ are awaitable. Coroutine Objects ----------------- -:term:`Coroutine` objects are :term:`awaitable` objects. +:term:`Coroutine objects ` are :term:`awaitable` objects. A coroutine's execution can be controlled by calling :meth:`__await__` and iterating over the result. When the coroutine has finished executing and returns, the iterator raises :exc:`StopIteration`, and the exception's diff --git a/Doc/reference/introduction.rst b/Doc/reference/introduction.rst index bb7e3906dba60..62480bd7dd9a6 100644 --- a/Doc/reference/introduction.rst +++ b/Doc/reference/introduction.rst @@ -93,7 +93,7 @@ Notation The descriptions of lexical analysis and syntax use a modified BNF grammar notation. This uses the following style of definition: -.. productionlist:: * +.. productionlist:: name: `lc_letter` (`lc_letter` | "_")* lc_letter: "a"..."z" diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index 685552f99f440..0d780e3ba8964 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -849,7 +849,7 @@ defines :meth:`__next__`, then :meth:`__iter__` can just return ``self``:: Generators ========== -:term:`Generator`\s are a simple and powerful tool for creating iterators. They +:term:`Generators ` are a simple and powerful tool for creating iterators. They are written like regular functions but use the :keyword:`yield` statement whenever they want to return data. Each time :func:`next` is called on it, the generator resumes where it left off (it remembers all the data values and which diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst index ca3eda05c515a..06bee9966c0be 100644 --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -2311,7 +2311,7 @@ Multi-threading =============== * The mechanism for serializing execution of concurrently running Python threads - (generally known as the :term:`GIL` or :term:`Global Interpreter Lock`) has + (generally known as the :term:`GIL` or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a "check interval" to allow thread diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index f1a033c6dae61..361e6db07c3cd 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -2309,9 +2309,9 @@ Encoders: :c:func:`PyUnicode_AsUTF8String` * :c:func:`PyUnicode_EncodeUTF32` * :c:func:`PyUnicode_EncodeUTF16` -* :c:func:`PyUnicode_EncodeUnicodeEscape:` use +* :c:func:`PyUnicode_EncodeUnicodeEscape` use :c:func:`PyUnicode_AsUnicodeEscapeString` -* :c:func:`PyUnicode_EncodeRawUnicodeEscape:` use +* :c:func:`PyUnicode_EncodeRawUnicodeEscape` use :c:func:`PyUnicode_AsRawUnicodeEscapeString` * :c:func:`PyUnicode_EncodeLatin1`: use :c:func:`PyUnicode_AsLatin1String` * :c:func:`PyUnicode_EncodeASCII`: use :c:func:`PyUnicode_AsASCIIString` diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst index b4540ac1dd902..1defee4090f28 100644 --- a/Doc/whatsnew/3.5.rst +++ b/Doc/whatsnew/3.5.rst @@ -412,7 +412,7 @@ uses were to provide type hints to function parameters and return values. It became evident that it would be beneficial for Python users, if the standard library included the base definitions and tools for type annotations. -:pep:`484` introduces a :term:`provisional module ` to +:pep:`484` introduces a :term:`provisional module ` to provide these standard definitions and tools, along with some conventions for situations where annotations are not available. @@ -726,7 +726,7 @@ New Modules typing ------ -The new :mod:`typing` :term:`provisional ` module +The new :mod:`typing` :term:`provisional ` module provides standard definitions and tools for function type annotations. See :ref:`Type Hints ` for more information. @@ -772,7 +772,7 @@ Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) asyncio ------- -Since the :mod:`asyncio` module is :term:`provisional `, +Since the :mod:`asyncio` module is :term:`provisional `, all changes introduced in Python 3.5 have also been backported to Python 3.4.x. Notable changes in the :mod:`asyncio` module since Python 3.4.0: @@ -1867,7 +1867,7 @@ A new :func:`~sys.set_coroutine_wrapper` function allows setting a global hook that will be called whenever a :term:`coroutine object ` is created by an :keyword:`async def` function. A corresponding :func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set -wrapper. Both functions are :term:`provisional `, +wrapper. Both functions are :term:`provisional `, and are intended for debugging purposes only. (Contributed by Yury Selivanov in :issue:`24017`.) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 04c1f7e71db32..85a6657fdfbda 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -1597,7 +1597,7 @@ to filter block traces by their address space (domain). typing ------ -Since the :mod:`typing` module is :term:`provisional `, +Since the :mod:`typing` module is :term:`provisional `, all changes introduced in Python 3.6 have also been backported to Python 3.5.x. diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 013f14dd9da16..4933cba3990b1 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -636,7 +636,7 @@ The :mod:`asyncio` module has received many new features, usability and :ref:`performance improvements `. Notable changes include: -* The new :term:`provisional ` :func:`asyncio.run` function can +* The new :term:`provisional ` :func:`asyncio.run` function can be used to run a coroutine from synchronous code by automatically creating and destroying the event loop. (Contributed by Yury Selivanov in :issue:`32314`.) diff --git a/Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst b/Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst new file mode 100644 index 0000000000000..152f6c98b9004 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-08-12-18-35-40.bpo-40204.C8A_pe.rst @@ -0,0 +1,3 @@ +Enable Sphinx 3.2 ``c_allow_pre_v3`` option and disable +``c_warn_on_allowed_pre_v3`` option to make the documentation compatible +with Sphinx 2 and Sphinx 3. From webhook-mailer at python.org Thu Aug 20 08:40:13 2020 From: webhook-mailer at python.org (Cleber Rosa) Date: Thu, 20 Aug 2020 12:40:13 -0000 Subject: [Python-checkins] bpo-41572: Fix grammar in BaseTransport.close docstring (GH-21914) Message-ID: https://github.com/python/cpython/commit/1afb42cfa82dad0ddd726f59c6c5fcb3962314db commit: 1afb42cfa82dad0ddd726f59c6c5fcb3962314db branch: master author: Cleber Rosa committer: GitHub date: 2020-08-20T18:10:01+05:30 summary: bpo-41572: Fix grammar in BaseTransport.close docstring (GH-21914) Fix grammar in BaseTransport.close docstring. https://bugs.python.org/issue41572 Signed-off-by: Cleber Rosa files: M Lib/asyncio/transports.py diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py index 513b1c024a4d6..45e155c94cad1 100644 --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -29,8 +29,8 @@ def close(self): Buffered data will be flushed asynchronously. No more data will be received. After all buffered data is flushed, the - protocol's connection_lost() method will (eventually) called - with None as its argument. + protocol's connection_lost() method will (eventually) be + called with None as its argument. """ raise NotImplementedError From webhook-mailer at python.org Thu Aug 20 16:08:55 2020 From: webhook-mailer at python.org (Mathieu Dupuy) Date: Thu, 20 Aug 2020 20:08:55 -0000 Subject: [Python-checkins] Doc: add a missing period (GH-21819) Message-ID: https://github.com/python/cpython/commit/12695f4c6d1167c0863098a586f3dfeb77a7cb9a commit: 12695f4c6d1167c0863098a586f3dfeb77a7cb9a branch: master author: Mathieu Dupuy committer: GitHub date: 2020-08-20T22:08:37+02:00 summary: Doc: add a missing period (GH-21819) files: M Doc/library/json.rst diff --git a/Doc/library/json.rst b/Doc/library/json.rst index b923c8e167052..e1a246aad6190 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -745,7 +745,7 @@ Command line options .. cmdoption:: --indent, --tab, --no-indent, --compact - Mutually exclusive options for whitespace control + Mutually exclusive options for whitespace control. .. versionadded:: 3.9 From webhook-mailer at python.org Fri Aug 21 08:20:06 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Fri, 21 Aug 2020 12:20:06 -0000 Subject: [Python-checkins] bpo-41572: Fix grammar in BaseTransport.close docstring (GH-21914) (GH-21930) Message-ID: https://github.com/python/cpython/commit/1370d9dd9fbd71e9d3c250c8e6644e0ee6534fca commit: 1370d9dd9fbd71e9d3c250c8e6644e0ee6534fca branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-21T17:49:58+05:30 summary: bpo-41572: Fix grammar in BaseTransport.close docstring (GH-21914) (GH-21930) Fix grammar in BaseTransport.close docstring. https://bugs.python.org/issue41572 Signed-off-by: Cleber Rosa (cherry picked from commit 1afb42cfa82dad0ddd726f59c6c5fcb3962314db) Co-authored-by: Cleber Rosa files: M Lib/asyncio/transports.py diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py index 513b1c024a4d6..45e155c94cad1 100644 --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -29,8 +29,8 @@ def close(self): Buffered data will be flushed asynchronously. No more data will be received. After all buffered data is flushed, the - protocol's connection_lost() method will (eventually) called - with None as its argument. + protocol's connection_lost() method will (eventually) be + called with None as its argument. """ raise NotImplementedError From webhook-mailer at python.org Fri Aug 21 17:34:14 2020 From: webhook-mailer at python.org (Ernest W. Durbin III) Date: Fri, 21 Aug 2020 21:34:14 -0000 Subject: [Python-checkins] Update references from travis-ci.org to travis-ci.com (GH-21919) Message-ID: https://github.com/python/cpython/commit/d2b0ce6c35dc18d3f81b1fa73579d2f65fbc85af commit: d2b0ce6c35dc18d3f81b1fa73579d2f65fbc85af branch: master author: Ernest W. Durbin III committer: GitHub date: 2020-08-21T14:34:06-07:00 summary: Update references from travis-ci.org to travis-ci.com (GH-21919) files: M README.rst diff --git a/README.rst b/README.rst index 96c9561e7c8eb..9a646a2ee5da2 100644 --- a/README.rst +++ b/README.rst @@ -1,9 +1,9 @@ This is Python version 3.10.0 alpha 0 ===================================== -.. image:: https://travis-ci.org/python/cpython.svg?branch=master +.. image:: https://travis-ci.com/python/cpython.svg?branch=master :alt: CPython build status on Travis CI - :target: https://travis-ci.org/python/cpython + :target: https://travis-ci.com/python/cpython .. image:: https://github.com/python/cpython/workflows/Tests/badge.svg :alt: CPython build status on GitHub Actions From webhook-mailer at python.org Fri Aug 21 18:29:02 2020 From: webhook-mailer at python.org (wyz23x2) Date: Fri, 21 Aug 2020 22:29:02 -0000 Subject: [Python-checkins] bpo-41573: Update release versions in General FAQ (GH-21915) Message-ID: https://github.com/python/cpython/commit/7173fc84e61b80b19261e47fca38030206a3a78e commit: 7173fc84e61b80b19261e47fca38030206a3a78e branch: master author: wyz23x2 <52805709+wyz23x2 at users.noreply.github.com> committer: GitHub date: 2020-08-21T18:28:54-04:00 summary: bpo-41573: Update release versions in General FAQ (GH-21915) files: M Doc/faq/general.rst diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst index eee3c3c203efa..cf70f16c6fe32 100644 --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -142,9 +142,9 @@ to fix critical bugs. Alpha, beta and release candidate versions have an additional suffix. The suffix for an alpha version is "aN" for some small number N, the suffix for a beta version is "bN" for some small number N, and the suffix for a release -candidate version is "cN" for some small number N. In other words, all versions +candidate version is "rcN" for some small number N. In other words, all versions labeled 2.0aN precede the versions labeled 2.0bN, which precede versions labeled -2.0cN, and *those* precede 2.0. +2.0rcN, and *those* precede 2.0. You may also find version numbers with a "+" suffix, e.g. "2.2+". These are unreleased versions, built directly from the CPython development repository. In @@ -309,8 +309,8 @@ releases. The latest stable releases can always be found on the `Python download page `_. There are two production-ready versions of Python: 2.x and 3.x. The recommended version is 3.x, which is supported by -most widely used libraries. Although 2.x is still widely used, `it will not -be maintained after January 1, 2020 `_. +most widely used libraries. Although 2.x is still widely used, `it is not +maintained anymore `_. How many people are using Python? --------------------------------- From webhook-mailer at python.org Fri Aug 21 18:29:38 2020 From: webhook-mailer at python.org (Andre Delfino) Date: Fri, 21 Aug 2020 22:29:38 -0000 Subject: [Python-checkins] Document vars behavior when __dict__ is missing (#21466) Message-ID: https://github.com/python/cpython/commit/802726acf6048338394a6a4750835c2cdd6a947b commit: 802726acf6048338394a6a4750835c2cdd6a947b branch: master author: Andre Delfino committer: GitHub date: 2020-08-21T19:29:34-03:00 summary: Document vars behavior when __dict__ is missing (#21466) files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 43c47c1da9434..d381d43d8e99c 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1734,6 +1734,9 @@ are always available. They are listed here in alphabetical order. locals dictionary is only useful for reads since updates to the locals dictionary are ignored. + A :exc:`TypeError` exception is raised if an object is specified but + it doesn't have a :attr:`~object.__dict__` attribute (for example, if + its class defines the :attr:`~object.__slots__` attribute). .. function:: zip(*iterables, strict=False) From webhook-mailer at python.org Fri Aug 21 23:28:56 2020 From: webhook-mailer at python.org (Larry Hastings) Date: Sat, 22 Aug 2020 03:28:56 -0000 Subject: [Python-checkins] Blurb release and pydoc topics for 3.5.10rc1. Message-ID: https://github.com/python/cpython/commit/45efe034e66973c8702b8bbbcac041c873aa5da3 commit: 45efe034e66973c8702b8bbbcac041c873aa5da3 branch: 3.5 author: Larry Hastings committer: Larry Hastings date: 2020-08-19T12:51:35-07:00 summary: Blurb release and pydoc topics for 3.5.10rc1. files: A Misc/NEWS.d/3.5.10rc1.rst D Misc/NEWS.d/next/Library/2020-03-25-16-02-16.bpo-39503.YmMbYn.rst D Misc/NEWS.d/next/Library/2020-07-01-16-59-46.bpo-41183.9stVAW.rst D Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst D Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst D Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst D Misc/NEWS.d/next/Security/2020-01-30-16-15-29.bpo-39503.B299Yq.rst D Misc/NEWS.d/next/Security/2020-03-14-14-57-44.bpo-38576.OowwQn.rst D Misc/NEWS.d/next/Security/2020-03-15-01-28-36.bpo-39073.6Szd3i.rst D Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst D Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst M Lib/pydoc_data/topics.py diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 1f001e34e6c74..3ddb0cf98767d 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Fri Nov 1 15:54:02 2019 +# Autogenerated by Sphinx on Wed Aug 19 12:50:32 2020 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' diff --git a/Misc/NEWS.d/3.5.10rc1.rst b/Misc/NEWS.d/3.5.10rc1.rst new file mode 100644 index 0000000000000..51c1aa32c8994 --- /dev/null +++ b/Misc/NEWS.d/3.5.10rc1.rst @@ -0,0 +1,106 @@ +.. bpo: 29778 +.. date: 2020-07-03-17-21-37 +.. nonce: cR_fGS +.. release date: 2020-08-19 +.. section: Security + +Ensure :file:`python3.dll` is loaded from correct locations when Python is +embedded (CVE-2020-15523). + +.. + +.. bpo: 41004 +.. date: 2020-06-29-16-02-29 +.. nonce: ovF0KZ +.. section: Security + +CVE-2020-14422: The __hash__() methods of ipaddress.IPv4Interface and +ipaddress.IPv6Interface incorrectly generated constant hash values of 32 and +128 respectively. This resulted in always causing hash collisions. The fix +uses hash() to generate hash values for the tuple of (address, mask length, +network address). + +.. + +.. bpo: 39073 +.. date: 2020-03-15-01-28-36 +.. nonce: 6Szd3i +.. section: Security + +Disallow CR or LF in email.headerregistry.Address arguments to guard against +header injection attacks. + +.. + +.. bpo: 38576 +.. date: 2020-03-14-14-57-44 +.. nonce: OowwQn +.. section: Security + +Disallow control characters in hostnames in http.client, addressing +CVE-2019-18348. Such potentially malicious header injection URLs now cause a +InvalidURL to be raised. + +.. + +.. bpo: 39503 +.. date: 2020-01-30-16-15-29 +.. nonce: B299Yq +.. section: Security + +CVE-2020-8492: The :class:`~urllib.request.AbstractBasicAuthHandler` class +of the :mod:`urllib.request` module uses an inefficient regular expression +which can be exploited by an attacker to cause a denial of service. Fix the +regex to prevent the catastrophic backtracking. Vulnerability reported by +Ben Caller and Matt Schwager. + +.. + +.. bpo: 38945 +.. date: 2019-12-01-22-44-40 +.. nonce: ztmNXc +.. section: Security + +Newline characters have been escaped when performing uu encoding to prevent +them from overflowing into to content section of the encoded file. This +prevents malicious or accidental modification of data during the decoding +process. + +.. + +.. bpo: 38804 +.. date: 2019-11-15-00-54-42 +.. nonce: vjbM8V +.. section: Security + +Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller. + +.. + +.. bpo: 39017 +.. date: 2020-07-12-22-16-58 +.. nonce: x3Cg-9 +.. section: Library + +Avoid infinite loop when reading specially crafted TAR files using the +tarfile module (CVE-2019-20907). + +.. + +.. bpo: 41183 +.. date: 2020-07-01-16-59-46 +.. nonce: 9stVAW +.. section: Library + +Use 3072 RSA keys and SHA-256 signature for test certs and keys. + +.. + +.. bpo: 39503 +.. date: 2020-03-25-16-02-16 +.. nonce: YmMbYn +.. section: Library + +:class:`~urllib.request.AbstractBasicAuthHandler` of :mod:`urllib.request` +now parses all WWW-Authenticate HTTP headers and accepts multiple challenges +per header: use the realm of the first Basic challenge. diff --git a/Misc/NEWS.d/next/Library/2020-03-25-16-02-16.bpo-39503.YmMbYn.rst b/Misc/NEWS.d/next/Library/2020-03-25-16-02-16.bpo-39503.YmMbYn.rst deleted file mode 100644 index be80ce79d91ed..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-03-25-16-02-16.bpo-39503.YmMbYn.rst +++ /dev/null @@ -1,3 +0,0 @@ -:class:`~urllib.request.AbstractBasicAuthHandler` of :mod:`urllib.request` -now parses all WWW-Authenticate HTTP headers and accepts multiple challenges -per header: use the realm of the first Basic challenge. diff --git a/Misc/NEWS.d/next/Library/2020-07-01-16-59-46.bpo-41183.9stVAW.rst b/Misc/NEWS.d/next/Library/2020-07-01-16-59-46.bpo-41183.9stVAW.rst deleted file mode 100644 index 1ca3c7d7996c1..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-01-16-59-46.bpo-41183.9stVAW.rst +++ /dev/null @@ -1 +0,0 @@ -Use 3072 RSA keys and SHA-256 signature for test certs and keys. diff --git a/Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst b/Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst deleted file mode 100644 index ad26676f8b856..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-12-22-16-58.bpo-39017.x3Cg-9.rst +++ /dev/null @@ -1 +0,0 @@ -Avoid infinite loop when reading specially crafted TAR files using the tarfile module (CVE-2019-20907). diff --git a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst b/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst deleted file mode 100644 index 1f45142d9f743..0000000000000 --- a/Misc/NEWS.d/next/Security/2019-11-15-00-54-42.bpo-38804.vjbM8V.rst +++ /dev/null @@ -1 +0,0 @@ -Fixes a ReDoS vulnerability in :mod:`http.cookiejar`. Patch by Ben Caller. diff --git a/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst b/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst deleted file mode 100644 index 1bf6ed567b241..0000000000000 --- a/Misc/NEWS.d/next/Security/2019-12-01-22-44-40.bpo-38945.ztmNXc.rst +++ /dev/null @@ -1 +0,0 @@ -Newline characters have been escaped when performing uu encoding to prevent them from overflowing into to content section of the encoded file. This prevents malicious or accidental modification of data during the decoding process. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Security/2020-01-30-16-15-29.bpo-39503.B299Yq.rst b/Misc/NEWS.d/next/Security/2020-01-30-16-15-29.bpo-39503.B299Yq.rst deleted file mode 100644 index 9f2800581ca5e..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-01-30-16-15-29.bpo-39503.B299Yq.rst +++ /dev/null @@ -1,5 +0,0 @@ -CVE-2020-8492: The :class:`~urllib.request.AbstractBasicAuthHandler` class of the -:mod:`urllib.request` module uses an inefficient regular expression which can -be exploited by an attacker to cause a denial of service. Fix the regex to -prevent the catastrophic backtracking. Vulnerability reported by Ben Caller -and Matt Schwager. diff --git a/Misc/NEWS.d/next/Security/2020-03-14-14-57-44.bpo-38576.OowwQn.rst b/Misc/NEWS.d/next/Security/2020-03-14-14-57-44.bpo-38576.OowwQn.rst deleted file mode 100644 index 1d03574651725..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-03-14-14-57-44.bpo-38576.OowwQn.rst +++ /dev/null @@ -1 +0,0 @@ -Disallow control characters in hostnames in http.client, addressing CVE-2019-18348. Such potentially malicious header injection URLs now cause a InvalidURL to be raised. diff --git a/Misc/NEWS.d/next/Security/2020-03-15-01-28-36.bpo-39073.6Szd3i.rst b/Misc/NEWS.d/next/Security/2020-03-15-01-28-36.bpo-39073.6Szd3i.rst deleted file mode 100644 index 6c9447b897bf6..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-03-15-01-28-36.bpo-39073.6Szd3i.rst +++ /dev/null @@ -1 +0,0 @@ -Disallow CR or LF in email.headerregistry.Address arguments to guard against header injection attacks. diff --git a/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst b/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst deleted file mode 100644 index f5a9db52fff52..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-06-29-16-02-29.bpo-41004.ovF0KZ.rst +++ /dev/null @@ -1 +0,0 @@ -CVE-2020-14422: The __hash__() methods of ipaddress.IPv4Interface and ipaddress.IPv6Interface incorrectly generated constant hash values of 32 and 128 respectively. This resulted in always causing hash collisions. The fix uses hash() to generate hash values for the tuple of (address, mask length, network address). diff --git a/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst b/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst deleted file mode 100644 index 998ffb1ee6667..0000000000000 --- a/Misc/NEWS.d/next/Security/2020-07-03-17-21-37.bpo-29778.cR_fGS.rst +++ /dev/null @@ -1,2 +0,0 @@ -Ensure :file:`python3.dll` is loaded from correct locations when Python is -embedded (CVE-2020-15523). From webhook-mailer at python.org Sat Aug 22 05:06:40 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 22 Aug 2020 09:06:40 -0000 Subject: [Python-checkins] bpo-40994: Ungroup items in collections.abc documentation for improved clarity (GH-21880) (#21927) Message-ID: https://github.com/python/cpython/commit/0694b82381ff27e10bb15172da0832a7e65aaa2d commit: 0694b82381ff27e10bb15172da0832a7e65aaa2d branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-22T05:06:14-04:00 summary: bpo-40994: Ungroup items in collections.abc documentation for improved clarity (GH-21880) (#21927) Use a less surprising document structure. Automerge-Triggered-By: @csabella (cherry picked from commit 2ce39631f679e14132a54dc90ce764259d26e166) Co-authored-by: Sydney Pemberton <46042811+sydneypemberton1986 at users.noreply.github.com> files: M Doc/library/collections.abc.rst diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index dc7ae30b6d2fa..db0e25bb0772e 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -98,12 +98,20 @@ ABC Inherits from Abstract Methods Mixin .. class:: Container - Hashable - Sized - Callable - ABCs for classes that provide respectively the methods :meth:`__contains__`, - :meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`. + ABC for classes that provide the :meth:`__contains__` method. + +.. class:: Hashable + + ABC for classes that provide the :meth:`__hash__` method. + +.. class:: Sized + + ABC for classes that provide the :meth:`__len__` method. + +.. class:: Callable + + ABC for classes that provide the :meth:`__call__` method. .. class:: Iterable From webhook-mailer at python.org Sat Aug 22 05:07:52 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 22 Aug 2020 09:07:52 -0000 Subject: [Python-checkins] Fix grammar in Doc/tutorial/controlflow.rst (GH-21885) (#21923) Message-ID: https://github.com/python/cpython/commit/e94d5db5125059b253c2fa8abddb2228053be9e8 commit: e94d5db5125059b253c2fa8abddb2228053be9e8 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-22T05:07:43-04:00 summary: Fix grammar in Doc/tutorial/controlflow.rst (GH-21885) (#21923) Automerge-Triggered-By: @csabella (cherry picked from commit 0be7c216e16f0d459f1c8f6209734c9b2b82fbd4) Co-authored-by: Denis Ovsienko files: M Doc/tutorial/controlflow.rst diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index de2c73a398eda..547f4aec47b68 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -658,7 +658,7 @@ Finally, consider this function definition which has a potential collision betwe return 'name' in kwds There is no possible call that will make it return ``True`` as the keyword ``'name'`` -will always to bind to the first parameter. For example:: +will always bind to the first parameter. For example:: >>> foo(1, **{'name': 2}) Traceback (most recent call last): From webhook-mailer at python.org Sat Aug 22 15:15:36 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 22 Aug 2020 19:15:36 -0000 Subject: [Python-checkins] Document vars behavior when __dict__ is missing (GH-21466) (GH-21941) Message-ID: https://github.com/python/cpython/commit/08045391a7aa87d4fbd3e8ef4c852c2fa4e81a8a commit: 08045391a7aa87d4fbd3e8ef4c852c2fa4e81a8a branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-22T16:15:29-03:00 summary: Document vars behavior when __dict__ is missing (GH-21466) (GH-21941) (cherry picked from commit 802726acf6048338394a6a4750835c2cdd6a947b) Co-authored-by: Andre Delfino files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 8cf755a0e6b38..c31efa96a92c1 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1714,6 +1714,9 @@ are always available. They are listed here in alphabetical order. locals dictionary is only useful for reads since updates to the locals dictionary are ignored. + A :exc:`TypeError` exception is raised if an object is specified but + it doesn't have a :attr:`~object.__dict__` attribute (for example, if + its class defines the :attr:`~object.__slots__` attribute). .. function:: zip(*iterables) From webhook-mailer at python.org Mon Aug 24 11:00:36 2020 From: webhook-mailer at python.org (Will Binns) Date: Mon, 24 Aug 2020 15:00:36 -0000 Subject: [Python-checkins] README: Add link to LICENSE (GH-21565) Message-ID: https://github.com/python/cpython/commit/3112aab31426f19a939f10cf114ff4137ce96624 commit: 3112aab31426f19a939f10cf114ff4137ce96624 branch: master author: Will Binns committer: GitHub date: 2020-08-24T12:00:12-03:00 summary: README: Add link to LICENSE (GH-21565) Thanks @wbnns for your contribution. files: M README.rst diff --git a/README.rst b/README.rst index 9a646a2ee5da2..14f4f32bca796 100644 --- a/README.rst +++ b/README.rst @@ -259,8 +259,9 @@ rights reserved. Copyright (c) 1991-1995 Stichting Mathematisch Centrum. All rights reserved. -See the file "LICENSE" for information on the history of this software, terms & -conditions for usage, and a DISCLAIMER OF ALL WARRANTIES. +See the `LICENSE `_ for +information on the history of this software, terms & conditions for usage, and a +DISCLAIMER OF ALL WARRANTIES. This Python distribution contains *no* GNU General Public License (GPL) code, so it may be used in proprietary projects. There are interfaces to some GNU From webhook-mailer at python.org Mon Aug 24 20:40:33 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Tue, 25 Aug 2020 00:40:33 -0000 Subject: [Python-checkins] bpo-41513: More accurate hypot() (GH-21916) Message-ID: https://github.com/python/cpython/commit/8e19c8be87017f6bef8e4c936b1e6ddacb558ad2 commit: 8e19c8be87017f6bef8e4c936b1e6ddacb558ad2 branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-24T17:40:08-07:00 summary: bpo-41513: More accurate hypot() (GH-21916) files: A Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst M Modules/mathmodule.c diff --git a/Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst b/Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst new file mode 100644 index 0000000000000..b4d0d9b63cf87 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-23-14-23-18.bpo-41513.DGqc_I.rst @@ -0,0 +1,3 @@ +Improved the accuracy of math.hypot(). Internally, each step is computed +with extra precision so that the result is now almost always correctly +rounded. diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 489802cc36745..1d6174132814b 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2404,52 +2404,79 @@ math_fmod_impl(PyObject *module, double x, double y) } /* -Given an *n* length *vec* of values and a value *max*, compute: +Given a *vec* of values, compute the vector norm: - sqrt(sum((x * scale) ** 2 for x in vec)) / scale + sqrt(sum(x ** 2 for x in vec)) - where scale is the first power of two - greater than max. - -or compute: - - max * sqrt(sum((x / max) ** 2 for x in vec)) - -The value of the *max* variable must be non-negative and -equal to the absolute value of the largest magnitude -entry in the vector. If n==0, then *max* should be 0.0. +The *max* variable should be equal to the largest fabs(x). +The *n* variable is the length of *vec*. +If n==0, then *max* should be 0.0. If an infinity is present in the vec, *max* should be INF. - The *found_nan* variable indicates whether some member of the *vec* is a NaN. -To improve accuracy and to increase the number of cases where -vector_norm() is commutative, we use a variant of Neumaier -summation specialized to exploit that we always know that -|csum| >= |x|. - -The *csum* variable tracks the cumulative sum and *frac* tracks -the cumulative fractional errors at each step. Since this -variant assumes that |csum| >= |x| at each step, we establish -the precondition by starting the accumulation from 1.0 which -represents the largest possible value of (x*scale)**2 or (x/max)**2. - -After the loop is finished, the initial 1.0 is subtracted out -for a net zero effect on the final sum. Since *csum* will be -greater than 1.0, the subtraction of 1.0 will not cause -fractional digits to be dropped from *csum*. - -To get the full benefit from compensated summation, the -largest addend should be in the range: 0.5 <= x <= 1.0. -Accordingly, scaling or division by *max* should not be skipped -even if not otherwise needed to prevent overflow or loss of precision. +To avoid overflow/underflow and to achieve high accuracy giving results +that are almost always correctly rounded, four techniques are used: + +* lossless scaling using a power-of-two scaling factor +* accurate squaring using Veltkamp-Dekker splitting +* compensated summation using a variant of the Neumaier algorithm +* differential correction of the square root + +The usual presentation of the Neumaier summation algorithm has an +expensive branch depending on which operand has the larger +magnitude. We avoid this cost by arranging the calculation so that +fabs(csum) is always as large as fabs(x). + +To establish the invariant, *csum* is initialized to 1.0 which is +always larger than x**2 after scaling or division by *max*. +After the loop is finished, the initial 1.0 is subtracted out for a +net zero effect on the final sum. Since *csum* will be greater than +1.0, the subtraction of 1.0 will not cause fractional digits to be +dropped from *csum*. + +To get the full benefit from compensated summation, the largest +addend should be in the range: 0.5 <= |x| <= 1.0. Accordingly, +scaling or division by *max* should not be skipped even if not +otherwise needed to prevent overflow or loss of precision. + +The assertion that hi*hi >= 1.0 is a bit subtle. Each vector element +gets scaled to a magnitude below 1.0. The Veltkamp-Dekker splitting +algorithm gives a *hi* value that is correctly rounded to half +precision. When a value at or below 1.0 is correctly rounded, it +never goes above 1.0. And when values at or below 1.0 are squared, +they remain at or below 1.0, thus preserving the summation invariant. + +The square root differential correction is needed because a +correctly rounded square root of a correctly rounded sum of +squares can still be off by as much as one ulp. + +The differential correction starts with a value *x* that is +the difference between the square of *h*, the possibly inaccurately +rounded square root, and the accurately computed sum of squares. +The correction is the first order term of the Maclaurin series +expansion of sqrt(h**2 + x) == h + x/(2*h) + O(x**2). + +Essentially, this differential correction is equivalent to one +refinement step in the Newton divide-and-average square root +algorithm, effectively doubling the number of accurate bits. +This technique is used in Dekker's SQRT2 algorithm and again in +Borges' ALGORITHM 4 and 5. + +References: + +1. Veltkamp-Dekker splitting: http://csclub.uwaterloo.ca/~pbarfuss/dekker1971.pdf +2. Compensated summation: http://www.ti3.tu-harburg.de/paper/rump/Ru08b.pdf +3. Square root diffential correction: https://arxiv.org/pdf/1904.09481.pdf */ static inline double vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) { + const double T27 = 134217729.0; /* ldexp(1.0, 27)+1.0) */ double x, csum = 1.0, oldcsum, frac = 0.0, scale; + double t, hi, lo, h; int max_e; Py_ssize_t i; @@ -2470,15 +2497,62 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) for (i=0 ; i < n ; i++) { x = vec[i]; assert(Py_IS_FINITE(x) && fabs(x) <= max); + x *= scale; - x = x*x; + assert(fabs(x) < 1.0); + + t = x * T27; + hi = t - (t - x); + lo = x - hi; + assert(hi + lo == x); + + x = hi * hi; assert(x <= 1.0); - assert(csum >= x); + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = 2.0 * hi * lo; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = lo * lo; + assert(fabs(csum) >= fabs(x)); oldcsum = csum; csum += x; frac += (oldcsum - csum) + x; } - return sqrt(csum - 1.0 + frac) / scale; + h = sqrt(csum - 1.0 + frac); + + x = h; + t = x * T27; + hi = t - (t - x); + lo = x - hi; + assert (hi + lo == x); + + x = -hi * hi; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = -2.0 * hi * lo; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = -lo * lo; + assert(fabs(csum) >= fabs(x)); + oldcsum = csum; + csum += x; + frac += (oldcsum - csum) + x; + + x = csum - 1.0 + frac; + return (h + x / (2.0 * h)) / scale; } /* When max_e < -1023, ldexp(1.0, -max_e) overflows. So instead of multiplying by a scale, we just divide by *max*. @@ -2489,7 +2563,7 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) x /= max; x = x*x; assert(x <= 1.0); - assert(csum >= x); + assert(fabs(csum) >= fabs(x)); oldcsum = csum; csum += x; frac += (oldcsum - csum) + x; From webhook-mailer at python.org Wed Aug 26 12:42:52 2020 From: webhook-mailer at python.org (Elvis Pranskevichus) Date: Wed, 26 Aug 2020 16:42:52 -0000 Subject: [Python-checkins] bpo-37658: Fix asyncio.wait_for() to respect waited task status (#21894) Message-ID: https://github.com/python/cpython/commit/a2118a14627256197bddcf4fcecad4c264c1e39d commit: a2118a14627256197bddcf4fcecad4c264c1e39d branch: master author: Elvis Pranskevichus committer: GitHub date: 2020-08-26T09:42:45-07:00 summary: bpo-37658: Fix asyncio.wait_for() to respect waited task status (#21894) Currently, if `asyncio.wait_for()` itself is cancelled it will always raise `CancelledError` regardless if the underlying task is still running. This is similar to a race with the timeout, which is handled already. files: A Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 7ecec9638489d..8b05434f273b5 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -465,9 +465,12 @@ async def wait_for(fut, timeout, *, loop=None): try: await waiter except exceptions.CancelledError: - fut.remove_done_callback(cb) - fut.cancel() - raise + if fut.done(): + return fut.result() + else: + fut.remove_done_callback(cb) + fut.cancel() + raise if fut.done(): return fut.result() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index 511961c32005a..74fc1e4a42133 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1120,6 +1120,22 @@ def gen(): res = loop.run_until_complete(task) self.assertEqual(res, "ok") + def test_wait_for_cancellation_race_condition(self): + def gen(): + yield 0.1 + yield 0.1 + yield 0.1 + yield 0.1 + + loop = self.new_test_loop(gen) + + fut = self.new_future(loop) + loop.call_later(0.1, fut.set_result, "ok") + task = loop.create_task(asyncio.wait_for(fut, timeout=1)) + loop.call_later(0.1, task.cancel) + res = loop.run_until_complete(task) + self.assertEqual(res, "ok") + def test_wait_for_waits_for_task_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst new file mode 100644 index 0000000000000..694fbbbf346dc --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst @@ -0,0 +1,2 @@ +:meth:`asyncio.wait_for` now properly handles races between cancellation of +itself and the completion of the wrapped awaitable. From webhook-mailer at python.org Wed Aug 26 12:42:53 2020 From: webhook-mailer at python.org (Elvis Pranskevichus) Date: Wed, 26 Aug 2020 16:42:53 -0000 Subject: [Python-checkins] bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (#21895) Message-ID: https://github.com/python/cpython/commit/c517fc712105c8e5930cb42baaebdbe37fc3e15f commit: c517fc712105c8e5930cb42baaebdbe37fc3e15f branch: master author: Elvis Pranskevichus committer: GitHub date: 2020-08-26T09:42:22-07:00 summary: bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (#21895) When I was fixing bpo-32751 back in GH-7216 I missed the case when *timeout* is zero or negative. This takes care of that. Props to @aaliddell for noticing the inconsistency. files: A Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index c37f0e1387980..7ecec9638489d 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -445,8 +445,13 @@ async def wait_for(fut, timeout, *, loop=None): if fut.done(): return fut.result() - fut.cancel() - raise exceptions.TimeoutError() + await _cancel_and_wait(fut, loop=loop) + try: + fut.result() + except exceptions.CancelledError as exc: + raise exceptions.TimeoutError() from exc + else: + raise exceptions.TimeoutError() waiter = loop.create_future() timeout_handle = loop.call_later(timeout, _release_waiter, waiter) diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index f9db066ce8983..511961c32005a 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1131,6 +1131,9 @@ async def inner(): nonlocal task_done try: await asyncio.sleep(0.2) + except asyncio.CancelledError: + await asyncio.sleep(_EPSILON) + raise finally: task_done = True @@ -1145,6 +1148,34 @@ async def inner(): chained = cm.exception.__context__ self.assertEqual(type(chained), asyncio.CancelledError) + def test_wait_for_waits_for_task_cancellation_w_timeout_0(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + task_done = False + + async def foo(): + async def inner(): + nonlocal task_done + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(_EPSILON) + raise + finally: + task_done = True + + inner_task = self.new_task(loop, inner()) + await asyncio.sleep(_EPSILON) + await asyncio.wait_for(inner_task, timeout=0) + + with self.assertRaises(asyncio.TimeoutError) as cm: + loop.run_until_complete(foo()) + + self.assertTrue(task_done) + chained = cm.exception.__context__ + self.assertEqual(type(chained), asyncio.CancelledError) + def test_wait_for_reraises_exception_during_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst new file mode 100644 index 0000000000000..c172ce5d9e948 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst @@ -0,0 +1,3 @@ +When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now +wait until the cancellation is complete also in the case when *timeout* is +<= 0, like it does with positive timeouts. From webhook-mailer at python.org Wed Aug 26 13:22:39 2020 From: webhook-mailer at python.org (Dong-hee Na) Date: Wed, 26 Aug 2020 17:22:39 -0000 Subject: [Python-checkins] bpo-40077: Convert _operator to use PyType_FromSpec (GH-21954) Message-ID: https://github.com/python/cpython/commit/31967fd8d0220af84e2698172d1378bffc8cd851 commit: 31967fd8d0220af84e2698172d1378bffc8cd851 branch: master author: Dong-hee Na committer: GitHub date: 2020-08-27T02:22:27+09:00 summary: bpo-40077: Convert _operator to use PyType_FromSpec (GH-21954) files: A Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst M Modules/_operator.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst new file mode 100644 index 0000000000000..ee950010e6d13 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-25-22-43-33.bpo-40077.vcxSUa.rst @@ -0,0 +1 @@ +Convert :mod:`_operator` to use :c:func:`PyType_FromSpec`. diff --git a/Modules/_operator.c b/Modules/_operator.c index 8a54829e5bbcc..c9d38aa342366 100644 --- a/Modules/_operator.c +++ b/Modules/_operator.c @@ -3,6 +3,20 @@ #include "clinic/_operator.c.h" +typedef struct { + PyObject *itemgetter_type; + PyObject *attrgetter_type; + PyObject *methodcaller_type; +} _operator_state; + +static inline _operator_state* +get_operator_state(PyObject *module) +{ + void *state = PyModule_GetState(module); + assert(state != NULL); + return (_operator_state *)state; +} + /*[clinic input] module _operator [clinic start generated code]*/ @@ -942,8 +956,6 @@ typedef struct { Py_ssize_t index; // -1 unless *item* is a single non-negative integer index } itemgetterobject; -static PyTypeObject itemgetter_type; - /* AC 3.5: treats first argument as an iterable, otherwise uses *args */ static PyObject * itemgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -960,13 +972,15 @@ itemgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) if (nitems <= 1) { if (!PyArg_UnpackTuple(args, "itemgetter", 1, 1, &item)) return NULL; - } else + } else { item = args; - + } + _operator_state *state = PyType_GetModuleState(type); /* create itemgetterobject structure */ - ig = PyObject_GC_New(itemgetterobject, &itemgetter_type); - if (ig == NULL) + ig = PyObject_GC_New(itemgetterobject, (PyTypeObject *) state->itemgetter_type); + if (ig == NULL) { return NULL; + } Py_INCREF(item); ig->item = item; @@ -994,9 +1008,11 @@ itemgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void itemgetter_dealloc(itemgetterobject *ig) { + PyTypeObject *tp = Py_TYPE(ig); PyObject_GC_UnTrack(ig); Py_XDECREF(ig->item); - PyObject_GC_Del(ig); + tp->tp_free(ig); + Py_DECREF(tp); } static int @@ -1093,49 +1109,25 @@ Return a callable object that fetches the given item(s) from its operand.\n\ After f = itemgetter(2), the call f(r) returns r[2].\n\ After g = itemgetter(2, 5, 3), the call g(r) returns (r[2], r[5], r[3])"); -static PyTypeObject itemgetter_type = { - PyVarObject_HEAD_INIT(NULL, 0) - "operator.itemgetter", /* tp_name */ - sizeof(itemgetterobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)itemgetter_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)itemgetter_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - (ternaryfunc)itemgetter_call, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - itemgetter_doc, /* tp_doc */ - (traverseproc)itemgetter_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - itemgetter_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - itemgetter_new, /* tp_new */ - 0, /* tp_free */ +static PyType_Slot itemgetter_type_slots[] = { + {Py_tp_doc, (void *)itemgetter_doc}, + {Py_tp_dealloc, itemgetter_dealloc}, + {Py_tp_call, itemgetter_call}, + {Py_tp_traverse, itemgetter_traverse}, + {Py_tp_methods, itemgetter_methods}, + {Py_tp_new, itemgetter_new}, + {Py_tp_getattro, PyObject_GenericGetAttr}, + {Py_tp_repr, itemgetter_repr}, + {0, 0} }; +static PyType_Spec itemgetter_type_spec = { + .name = "operator.itemgetter", + .basicsize = sizeof(itemgetterobject), + .itemsize = 0, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .slots = itemgetter_type_slots, +}; /* attrgetter object **********************************************************/ @@ -1145,8 +1137,6 @@ typedef struct { PyObject *attr; } attrgetterobject; -static PyTypeObject attrgetter_type; - /* AC 3.5: treats first argument as an iterable, otherwise uses *args */ static PyObject * attrgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -1246,8 +1236,9 @@ attrgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } } + _operator_state *state = PyType_GetModuleState(type); /* create attrgetterobject structure */ - ag = PyObject_GC_New(attrgetterobject, &attrgetter_type); + ag = PyObject_GC_New(attrgetterobject, (PyTypeObject *)state->attrgetter_type); if (ag == NULL) { Py_DECREF(attr); return NULL; @@ -1263,9 +1254,11 @@ attrgetter_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void attrgetter_dealloc(attrgetterobject *ag) { + PyTypeObject *tp = Py_TYPE(ag); PyObject_GC_UnTrack(ag); Py_XDECREF(ag->attr); - PyObject_GC_Del(ag); + tp->tp_free(ag); + Py_DECREF(tp); } static int @@ -1438,47 +1431,24 @@ After g = attrgetter('name', 'date'), the call g(r) returns (r.name, r.date).\n\ After h = attrgetter('name.first', 'name.last'), the call h(r) returns\n\ (r.name.first, r.name.last)."); -static PyTypeObject attrgetter_type = { - PyVarObject_HEAD_INIT(NULL, 0) - "operator.attrgetter", /* tp_name */ - sizeof(attrgetterobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)attrgetter_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)attrgetter_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - (ternaryfunc)attrgetter_call, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - attrgetter_doc, /* tp_doc */ - (traverseproc)attrgetter_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - attrgetter_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - attrgetter_new, /* tp_new */ - 0, /* tp_free */ +static PyType_Slot attrgetter_type_slots[] = { + {Py_tp_doc, (void *)attrgetter_doc}, + {Py_tp_dealloc, attrgetter_dealloc}, + {Py_tp_call, attrgetter_call}, + {Py_tp_traverse, attrgetter_traverse}, + {Py_tp_methods, attrgetter_methods}, + {Py_tp_new, attrgetter_new}, + {Py_tp_getattro, PyObject_GenericGetAttr}, + {Py_tp_repr, attrgetter_repr}, + {0, 0} +}; + +static PyType_Spec attrgetter_type_spec = { + .name = "operator.attrgetter", + .basicsize = sizeof(attrgetterobject), + .itemsize = 0, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .slots = attrgetter_type_slots, }; @@ -1491,8 +1461,6 @@ typedef struct { PyObject *kwds; } methodcallerobject; -static PyTypeObject methodcaller_type; - /* AC 3.5: variable number of arguments, not currently support by AC */ static PyObject * methodcaller_new(PyTypeObject *type, PyObject *args, PyObject *kwds) @@ -1513,10 +1481,12 @@ methodcaller_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return NULL; } + _operator_state *state = PyType_GetModuleState(type); /* create methodcallerobject structure */ - mc = PyObject_GC_New(methodcallerobject, &methodcaller_type); - if (mc == NULL) + mc = PyObject_GC_New(methodcallerobject, (PyTypeObject *)state->methodcaller_type); + if (mc == NULL) { return NULL; + } name = PyTuple_GET_ITEM(args, 0); Py_INCREF(name); @@ -1539,11 +1509,13 @@ methodcaller_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void methodcaller_dealloc(methodcallerobject *mc) { + PyTypeObject *tp = Py_TYPE(mc); PyObject_GC_UnTrack(mc); Py_XDECREF(mc->name); Py_XDECREF(mc->args); Py_XDECREF(mc->kwds); - PyObject_GC_Del(mc); + tp->tp_free(mc); + Py_DECREF(tp); } static int @@ -1704,63 +1676,52 @@ After f = methodcaller('name'), the call f(r) returns r.name().\n\ After g = methodcaller('name', 'date', foo=1), the call g(r) returns\n\ r.name('date', foo=1)."); -static PyTypeObject methodcaller_type = { - PyVarObject_HEAD_INIT(NULL, 0) - "operator.methodcaller", /* tp_name */ - sizeof(methodcallerobject), /* tp_basicsize */ - 0, /* tp_itemsize */ - /* methods */ - (destructor)methodcaller_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)methodcaller_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - (ternaryfunc)methodcaller_call, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ - methodcaller_doc, /* tp_doc */ - (traverseproc)methodcaller_traverse, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - methodcaller_methods, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - methodcaller_new, /* tp_new */ - 0, /* tp_free */ +static PyType_Slot methodcaller_type_slots[] = { + {Py_tp_doc, (void *)methodcaller_doc}, + {Py_tp_dealloc, methodcaller_dealloc}, + {Py_tp_call, methodcaller_call}, + {Py_tp_traverse, methodcaller_traverse}, + {Py_tp_methods, methodcaller_methods}, + {Py_tp_new, methodcaller_new}, + {Py_tp_getattro, PyObject_GenericGetAttr}, + {Py_tp_repr, methodcaller_repr}, + {0, 0} }; +static PyType_Spec methodcaller_type_spec = { + .name = "operator.methodcaller", + .basicsize = sizeof(methodcallerobject), + .itemsize = 0, + .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .slots = methodcaller_type_slots, +}; static int operator_exec(PyObject *module) { - PyTypeObject *types[] = { - &itemgetter_type, - &attrgetter_type, - &methodcaller_type - }; + _operator_state *state = get_operator_state(module); + state->attrgetter_type = PyType_FromModuleAndSpec(module, &attrgetter_type_spec, NULL); + if (state->attrgetter_type == NULL) { + return -1; + } + if (PyModule_AddType(module, (PyTypeObject *)state->attrgetter_type) < 0) { + return -1; + } - for (size_t i = 0; i < Py_ARRAY_LENGTH(types); i++) { - if (PyModule_AddType(module, types[i]) < 0) { - return -1; - } + state->itemgetter_type = PyType_FromModuleAndSpec(module, &itemgetter_type_spec, NULL); + if (state->itemgetter_type == NULL) { + return -1; + } + if (PyModule_AddType(module, (PyTypeObject *)state->itemgetter_type) < 0) { + return -1; + } + + state->methodcaller_type = PyType_FromModuleAndSpec(module, &methodcaller_type_spec, NULL); + if (state->methodcaller_type == NULL) { + return -1; + } + if (PyModule_AddType(module, (PyTypeObject *)state->methodcaller_type) < 0) { + return -1; } return 0; @@ -1772,17 +1733,42 @@ static struct PyModuleDef_Slot operator_slots[] = { {0, NULL} }; +static int +operator_traverse(PyObject *module, visitproc visit, void *arg) +{ + _operator_state *state = get_operator_state(module); + Py_VISIT(state->attrgetter_type); + Py_VISIT(state->itemgetter_type); + Py_VISIT(state->methodcaller_type); + return 0; +} + +static int +operator_clear(PyObject *module) +{ + _operator_state *state = get_operator_state(module); + Py_CLEAR(state->attrgetter_type); + Py_CLEAR(state->itemgetter_type); + Py_CLEAR(state->methodcaller_type); + return 0; +} + +static void +operator_free(void *module) +{ + operator_clear((PyObject *)module); +} static struct PyModuleDef operatormodule = { PyModuleDef_HEAD_INIT, - "_operator", - operator_doc, - 0, - operator_methods, - operator_slots, - NULL, - NULL, - NULL + .m_name = "_operator", + .m_doc = operator_doc, + .m_size = sizeof(_operator_state), + .m_methods = operator_methods, + .m_slots = operator_slots, + .m_traverse = operator_traverse, + .m_clear = operator_clear, + .m_free = operator_free, }; PyMODINIT_FUNC From webhook-mailer at python.org Wed Aug 26 14:26:38 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Wed, 26 Aug 2020 18:26:38 -0000 Subject: [Python-checkins] bpo-37658: Fix asyncio.wait_for() to respect waited task status (GH-21894) (#21965) Message-ID: https://github.com/python/cpython/commit/6e1954cd8286e083e7f8d09516d91b6b15769a4e commit: 6e1954cd8286e083e7f8d09516d91b6b15769a4e branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-26T11:26:28-07:00 summary: bpo-37658: Fix asyncio.wait_for() to respect waited task status (GH-21894) (#21965) Currently, if `asyncio.wait_for()` itself is cancelled it will always raise `CancelledError` regardless if the underlying task is still running. This is similar to a race with the timeout, which is handled already. (cherry picked from commit a2118a14627256197bddcf4fcecad4c264c1e39d) Co-authored-by: Elvis Pranskevichus files: A Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 66e81f9200a36..373be37fb4bdd 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -475,9 +475,12 @@ async def wait_for(fut, timeout, *, loop=None): try: await waiter except exceptions.CancelledError: - fut.remove_done_callback(cb) - fut.cancel() - raise + if fut.done(): + return fut.result() + else: + fut.remove_done_callback(cb) + fut.cancel() + raise if fut.done(): return fut.result() diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index c21b06938fe3f..a70dc0a87e1c4 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -861,6 +861,22 @@ def gen(): res = loop.run_until_complete(task) self.assertEqual(res, "ok") + def test_wait_for_cancellation_race_condition(self): + def gen(): + yield 0.1 + yield 0.1 + yield 0.1 + yield 0.1 + + loop = self.new_test_loop(gen) + + fut = self.new_future(loop) + loop.call_later(0.1, fut.set_result, "ok") + task = loop.create_task(asyncio.wait_for(fut, timeout=1)) + loop.call_later(0.1, task.cancel) + res = loop.run_until_complete(task) + self.assertEqual(res, "ok") + def test_wait_for_waits_for_task_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst new file mode 100644 index 0000000000000..694fbbbf346dc --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-21-40.bpo-37658.f9nivB.rst @@ -0,0 +1,2 @@ +:meth:`asyncio.wait_for` now properly handles races between cancellation of +itself and the completion of the wrapped awaitable. From webhook-mailer at python.org Wed Aug 26 16:10:09 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Wed, 26 Aug 2020 20:10:09 -0000 Subject: [Python-checkins] Fix typos in comment (GH-21966) Message-ID: https://github.com/python/cpython/commit/82e79480d6e61940d7007d9026fbff0b1a11ad9a commit: 82e79480d6e61940d7007d9026fbff0b1a11ad9a branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-26T13:09:40-07:00 summary: Fix typos in comment (GH-21966) files: M Modules/mathmodule.c diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 1d6174132814b..1704d8efd31c3 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2440,7 +2440,7 @@ addend should be in the range: 0.5 <= |x| <= 1.0. Accordingly, scaling or division by *max* should not be skipped even if not otherwise needed to prevent overflow or loss of precision. -The assertion that hi*hi >= 1.0 is a bit subtle. Each vector element +The assertion that hi*hi <= 1.0 is a bit subtle. Each vector element gets scaled to a magnitude below 1.0. The Veltkamp-Dekker splitting algorithm gives a *hi* value that is correctly rounded to half precision. When a value at or below 1.0 is correctly rounded, it @@ -2458,7 +2458,7 @@ The correction is the first order term of the Maclaurin series expansion of sqrt(h**2 + x) == h + x/(2*h) + O(x**2). Essentially, this differential correction is equivalent to one -refinement step in the Newton divide-and-average square root +refinement step in Newton's divide-and-average square root algorithm, effectively doubling the number of accurate bits. This technique is used in Dekker's SQRT2 algorithm and again in Borges' ALGORITHM 4 and 5. From webhook-mailer at python.org Wed Aug 26 16:59:40 2020 From: webhook-mailer at python.org (Elvis Pranskevichus) Date: Wed, 26 Aug 2020 20:59:40 -0000 Subject: [Python-checkins] [3.8] bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (GH-21895) (#21967) Message-ID: https://github.com/python/cpython/commit/57b698886b47bb81c782c2ba80a8a72fe66c7aad commit: 57b698886b47bb81c782c2ba80a8a72fe66c7aad branch: 3.8 author: Elvis Pranskevichus committer: GitHub date: 2020-08-26T13:59:17-07:00 summary: [3.8] bpo-32751: Wait for task cancel in asyncio.wait_for() when timeout <= 0 (GH-21895) (#21967) When I was fixing bpo-32751 back in GH-7216 I missed the case when *timeout* is zero or negative. This takes care of that. Props to @aaliddell for noticing the inconsistency.. (cherry picked from commit c517fc712105c8e5930cb42baaebdbe37fc3e15f) files: A Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst M Lib/asyncio/tasks.py M Lib/test/test_asyncio/test_tasks.py diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py index 373be37fb4bdd..9ca9fa0a94a93 100644 --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -460,8 +460,13 @@ async def wait_for(fut, timeout, *, loop=None): if fut.done(): return fut.result() - fut.cancel() - raise exceptions.TimeoutError() + await _cancel_and_wait(fut, loop=loop) + try: + fut.result() + except exceptions.CancelledError as exc: + raise exceptions.TimeoutError() from exc + else: + raise exceptions.TimeoutError() waiter = loop.create_future() timeout_handle = loop.call_later(timeout, _release_waiter, waiter) diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index a70dc0a87e1c4..c402f8f021f3a 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -888,6 +888,9 @@ async def inner(): nonlocal task_done try: await asyncio.sleep(0.2) + except asyncio.CancelledError: + await asyncio.sleep(0.1) + raise finally: task_done = True @@ -900,6 +903,34 @@ async def inner(): loop.run_until_complete(foo()) + def test_wait_for_waits_for_task_cancellation_w_timeout_0(self): + loop = asyncio.new_event_loop() + self.addCleanup(loop.close) + + task_done = False + + async def foo(): + async def inner(): + nonlocal task_done + try: + await asyncio.sleep(10) + except asyncio.CancelledError: + await asyncio.sleep(0.1) + raise + finally: + task_done = True + + inner_task = self.new_task(loop, inner()) + await asyncio.sleep(0.1) + await asyncio.wait_for(inner_task, timeout=0) + + with self.assertRaises(asyncio.TimeoutError) as cm: + loop.run_until_complete(foo()) + + self.assertTrue(task_done) + chained = cm.exception.__context__ + self.assertEqual(type(chained), asyncio.CancelledError) + def test_wait_for_self_cancellation(self): loop = asyncio.new_event_loop() self.addCleanup(loop.close) diff --git a/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst new file mode 100644 index 0000000000000..c172ce5d9e948 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-15-50-12.bpo-32751.85je5X.rst @@ -0,0 +1,3 @@ +When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now +wait until the cancellation is complete also in the case when *timeout* is +<= 0, like it does with positive timeouts. From webhook-mailer at python.org Wed Aug 26 20:43:13 2020 From: webhook-mailer at python.org (MingZhe Hu) Date: Thu, 27 Aug 2020 00:43:13 -0000 Subject: [Python-checkins] bpo-41624: fix documentation of typing.Coroutine (GH-21952) Message-ID: https://github.com/python/cpython/commit/8c58d2a216ca2b5965361df9b8d8944bc7d4854d commit: 8c58d2a216ca2b5965361df9b8d8944bc7d4854d branch: master author: MingZhe Hu committer: GitHub date: 2020-08-27T02:42:37+02:00 summary: bpo-41624: fix documentation of typing.Coroutine (GH-21952) files: A Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 8208680669de6..9f98f8ce3f642 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1365,7 +1365,7 @@ Corresponding to other types in :mod:`collections.abc` Asynchronous programming """""""""""""""""""""""" -.. class:: Coroutine(Awaitable[V_co], Generic[T_co T_contra, V_co]) +.. class:: Coroutine(Awaitable[V_co], Generic[T_co, T_contra, V_co]) A generic version of :class:`collections.abc.Coroutine`. The variance and order of type variables diff --git a/Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst b/Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst new file mode 100644 index 0000000000000..bdbc5a445f339 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst @@ -0,0 +1 @@ +Fix the signature of :class:`typing.Coroutine`. From webhook-mailer at python.org Wed Aug 26 20:47:20 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Thu, 27 Aug 2020 00:47:20 -0000 Subject: [Python-checkins] bpo-33660: Fix PosixPath to resolve a relative path on root (GH-21975) Message-ID: https://github.com/python/cpython/commit/7475aa2c590e33a47f5e79e4079bca0645e93f2f commit: 7475aa2c590e33a47f5e79e4079bca0645e93f2f branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-27T02:47:10+02:00 summary: bpo-33660: Fix PosixPath to resolve a relative path on root (GH-21975) (cherry picked from commit 94ad6c674f7687ef22853cb8d42b440d6b42ddc8) Co-authored-by: Dong-hee Na files: A Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst M Lib/pathlib.py M Lib/test/test_pathlib.py diff --git a/Lib/pathlib.py b/Lib/pathlib.py index ff8bac94bc0be..4f72bab3bdbbc 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -329,7 +329,10 @@ def _resolve(path, rest): # parent dir path, _, _ = path.rpartition(sep) continue - newpath = path + sep + name + if path.endswith(sep): + newpath = path + name + else: + newpath = path + sep + name if newpath in seen: # Already seen this path path = seen[newpath] diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 36226948222d6..e9f928a986555 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -2214,6 +2214,15 @@ def test_open_mode(self): st = os.stat(join('other_new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o644) + def test_resolve_root(self): + current_directory = os.getcwd() + try: + os.chdir('/') + p = self.cls('spam') + self.assertEqual(str(p.resolve()), '/spam') + finally: + os.chdir(current_directory) + def test_touch_mode(self): old_mask = os.umask(0) self.addCleanup(os.umask, old_mask) diff --git a/Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst b/Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst new file mode 100644 index 0000000000000..cce3dbb1c6ea5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst @@ -0,0 +1,2 @@ +Fix pathlib.PosixPath to resolve a relative path located on the root +directory properly. From webhook-mailer at python.org Wed Aug 26 20:49:27 2020 From: webhook-mailer at python.org (Mason) Date: Thu, 27 Aug 2020 00:49:27 -0000 Subject: [Python-checkins] Add missing word (GH-21936) Message-ID: https://github.com/python/cpython/commit/b260635b3db5715adf7a9a3df542c171fb7296ef commit: b260635b3db5715adf7a9a3df542c171fb7296ef branch: master author: Mason <38963371+nosamproductions at users.noreply.github.com> committer: GitHub date: 2020-08-27T02:49:14+02:00 summary: Add missing word (GH-21936) "data to lost" -> "data to be lost" files: M Doc/library/multiprocessing.rst diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 28510acd52bd4..f8e7556f25b4e 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -861,7 +861,7 @@ For an example of the usage of queues for interprocess communication see A better name for this method might be ``allow_exit_without_flush()``. It is likely to cause enqueued - data to lost, and you almost certainly will not need to use it. + data to be lost, and you almost certainly will not need to use it. It is really only there if you need the current process to exit immediately without waiting to flush enqueued data to the underlying pipe, and you don't care about lost data. From webhook-mailer at python.org Wed Aug 26 20:51:17 2020 From: webhook-mailer at python.org (Irit Katriel) Date: Thu, 27 Aug 2020 00:51:17 -0000 Subject: [Python-checkins] bpo-41609: Fix output of pdb's whatis command for instance methods (GH-21935) Message-ID: https://github.com/python/cpython/commit/022bc7572f061e1d1132a4db9d085b29707701e7 commit: 022bc7572f061e1d1132a4db9d085b29707701e7 branch: master author: Irit Katriel committer: GitHub date: 2020-08-27T02:51:12+02:00 summary: bpo-41609: Fix output of pdb's whatis command for instance methods (GH-21935) files: A Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst M Lib/pdb.py M Lib/test/test_pdb.py diff --git a/Lib/pdb.py b/Lib/pdb.py index 081023526c0ea..d7d957159458b 100755 --- a/Lib/pdb.py +++ b/Lib/pdb.py @@ -1312,21 +1312,21 @@ def do_whatis(self, arg): # _getval() already printed the error return code = None - # Is it a function? + # Is it an instance method? try: - code = value.__code__ + code = value.__func__.__code__ except Exception: pass if code: - self.message('Function %s' % code.co_name) + self.message('Method %s' % code.co_name) return - # Is it an instance method? + # Is it a function? try: - code = value.__func__.__code__ + code = value.__code__ except Exception: pass if code: - self.message('Method %s' % code.co_name) + self.message('Function %s' % code.co_name) return # Is it a class? if value.__class__ is type: diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index 1a2bbb382e864..e56451360d643 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -425,6 +425,47 @@ def test_list_commands(): (Pdb) continue """ +def test_pdb_whatis_command(): + """Test the whatis command + + >>> myvar = (1,2) + >>> def myfunc(): + ... pass + + >>> class MyClass: + ... def mymethod(self): + ... pass + + >>> def test_function(): + ... import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace() + + >>> with PdbTestInput([ # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE + ... 'whatis myvar', + ... 'whatis myfunc', + ... 'whatis MyClass', + ... 'whatis MyClass()', + ... 'whatis MyClass.mymethod', + ... 'whatis MyClass().mymethod', + ... 'continue', + ... ]): + ... test_function() + --Return-- + > (2)test_function()->None + -> import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace() + (Pdb) whatis myvar + + (Pdb) whatis myfunc + Function myfunc + (Pdb) whatis MyClass + Class test.test_pdb.MyClass + (Pdb) whatis MyClass() + + (Pdb) whatis MyClass.mymethod + Function mymethod + (Pdb) whatis MyClass().mymethod + Method mymethod + (Pdb) continue + """ def test_post_mortem(): """Test post mortem traceback debugging. diff --git a/Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst b/Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst new file mode 100644 index 0000000000000..ecaf40eee7bab --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst @@ -0,0 +1 @@ +The pdb whatis command correctly reports instance methods as 'Method' rather than 'Function'. \ No newline at end of file From webhook-mailer at python.org Wed Aug 26 21:17:38 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Thu, 27 Aug 2020 01:17:38 -0000 Subject: [Python-checkins] bpo-41609: Fix output of pdb's whatis command for instance methods (GH-21935) (#21976) Message-ID: https://github.com/python/cpython/commit/641279e6e51b5d2e10d3fbffe6330e47c94c4bb2 commit: 641279e6e51b5d2e10d3fbffe6330e47c94c4bb2 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-27T03:17:05+02:00 summary: bpo-41609: Fix output of pdb's whatis command for instance methods (GH-21935) (#21976) (cherry picked from commit 022bc7572f061e1d1132a4db9d085b29707701e7) Co-authored-by: Irit Katriel files: A Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst M Lib/pdb.py M Lib/test/test_pdb.py diff --git a/Lib/pdb.py b/Lib/pdb.py index 081023526c0ea..d7d957159458b 100755 --- a/Lib/pdb.py +++ b/Lib/pdb.py @@ -1312,21 +1312,21 @@ def do_whatis(self, arg): # _getval() already printed the error return code = None - # Is it a function? + # Is it an instance method? try: - code = value.__code__ + code = value.__func__.__code__ except Exception: pass if code: - self.message('Function %s' % code.co_name) + self.message('Method %s' % code.co_name) return - # Is it an instance method? + # Is it a function? try: - code = value.__func__.__code__ + code = value.__code__ except Exception: pass if code: - self.message('Method %s' % code.co_name) + self.message('Function %s' % code.co_name) return # Is it a class? if value.__class__ is type: diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index 0e7ae1d86ed80..9c9471a8cc8df 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -425,6 +425,47 @@ def test_list_commands(): (Pdb) continue """ +def test_pdb_whatis_command(): + """Test the whatis command + + >>> myvar = (1,2) + >>> def myfunc(): + ... pass + + >>> class MyClass: + ... def mymethod(self): + ... pass + + >>> def test_function(): + ... import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace() + + >>> with PdbTestInput([ # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE + ... 'whatis myvar', + ... 'whatis myfunc', + ... 'whatis MyClass', + ... 'whatis MyClass()', + ... 'whatis MyClass.mymethod', + ... 'whatis MyClass().mymethod', + ... 'continue', + ... ]): + ... test_function() + --Return-- + > (2)test_function()->None + -> import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace() + (Pdb) whatis myvar + + (Pdb) whatis myfunc + Function myfunc + (Pdb) whatis MyClass + Class test.test_pdb.MyClass + (Pdb) whatis MyClass() + + (Pdb) whatis MyClass.mymethod + Function mymethod + (Pdb) whatis MyClass().mymethod + Method mymethod + (Pdb) continue + """ def test_post_mortem(): """Test post mortem traceback debugging. diff --git a/Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst b/Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst new file mode 100644 index 0000000000000..ecaf40eee7bab --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-21-15-51-15.bpo-41609.JmiUKG.rst @@ -0,0 +1 @@ +The pdb whatis command correctly reports instance methods as 'Method' rather than 'Function'. \ No newline at end of file From webhook-mailer at python.org Thu Aug 27 01:45:59 2020 From: webhook-mailer at python.org (wmeehan) Date: Thu, 27 Aug 2020 05:45:59 -0000 Subject: [Python-checkins] bpo-41524: fix pointer bug in PyOS_mystr{n}icmp (GH-21845) Message-ID: https://github.com/python/cpython/commit/97eaf2b5e5c826b9abe59896a363853bef55c5d9 commit: 97eaf2b5e5c826b9abe59896a363853bef55c5d9 branch: master author: wmeehan committer: GitHub date: 2020-08-27T14:45:25+09:00 summary: bpo-41524: fix pointer bug in PyOS_mystr{n}icmp (GH-21845) * bpo-41524: fix pointer bug in PyOS_mystr{n}icmp The existing implementations of PyOS_mystrnicmp and PyOS_mystricmp can increment pointers beyond the end of a string. This commit fixes those cases by moving the mutation out of the condition. * ?? Added by blurb_it. * Address comments Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> files: A Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst M Python/pystrcmp.c diff --git a/Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst b/Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst new file mode 100644 index 0000000000000..4704e29be29bb --- /dev/null +++ b/Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst @@ -0,0 +1,2 @@ +Fix bug in PyOS_mystrnicmp and PyOS_mystricmp that incremented +pointers beyond the end of a string. \ No newline at end of file diff --git a/Python/pystrcmp.c b/Python/pystrcmp.c index f9c2277cb56dc..9224ce4c70605 100644 --- a/Python/pystrcmp.c +++ b/Python/pystrcmp.c @@ -6,21 +6,25 @@ int PyOS_mystrnicmp(const char *s1, const char *s2, Py_ssize_t size) { + const unsigned char *p1, *p2; if (size == 0) return 0; - while ((--size > 0) && - (tolower((unsigned)*s1) == tolower((unsigned)*s2))) { - if (!*s1++ || !*s2++) - break; + p1 = (const unsigned char *)s1; + p2 = (const unsigned char *)s2; + for (; (--size > 0) && *p1 && *p2 && (tolower(*p1) == tolower(*p2)); + p1++, p2++) { + ; } - return tolower((unsigned)*s1) - tolower((unsigned)*s2); + return tolower(*p1) - tolower(*p2); } int PyOS_mystricmp(const char *s1, const char *s2) { - while (*s1 && (tolower((unsigned)*s1++) == tolower((unsigned)*s2++))) { + const unsigned char *p1 = (const unsigned char *)s1; + const unsigned char *p2 = (const unsigned char *)s2; + for (; *p1 && *p2 && (tolower(*p1) == tolower(*p2)); p1++, p2++) { ; } - return (tolower((unsigned)*s1) - tolower((unsigned)*s2)); + return (tolower(*p1) - tolower(*p2)); } From webhook-mailer at python.org Thu Aug 27 07:28:20 2020 From: webhook-mailer at python.org (Zackery Spytz) Date: Thu, 27 Aug 2020 11:28:20 -0000 Subject: [Python-checkins] bpo-41634: Fix a typo in the curses documentation (GH-21958) Message-ID: https://github.com/python/cpython/commit/398575c210f79627830c5c470184f54ace950ac6 commit: 398575c210f79627830c5c470184f54ace950ac6 branch: master author: Zackery Spytz committer: GitHub date: 2020-08-27T16:58:16+05:30 summary: bpo-41634: Fix a typo in the curses documentation (GH-21958) files: M Doc/library/curses.rst diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 0b687db1bd2c4..3684d54d4ecad 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -717,7 +717,7 @@ the following methods and attributes: window.addch(y, x, ch[, attr]) Paint character *ch* at ``(y, x)`` with attributes *attr*, overwriting any - character previously painter at that location. By default, the character + character previously painted at that location. By default, the character position and attributes are the current settings for the window object. .. note:: From webhook-mailer at python.org Thu Aug 27 09:37:06 2020 From: webhook-mailer at python.org (Petr Viktorin) Date: Thu, 27 Aug 2020 13:37:06 -0000 Subject: [Python-checkins] bpo-38787: Clarify docs for PyType_GetModule and warn against common mistake (GH-20215) Message-ID: https://github.com/python/cpython/commit/d9a966ae08258da2ce2a432c943d8194760f09c4 commit: d9a966ae08258da2ce2a432c943d8194760f09c4 branch: master author: Petr Viktorin committer: GitHub date: 2020-08-27T15:36:48+02:00 summary: bpo-38787: Clarify docs for PyType_GetModule and warn against common mistake (GH-20215) files: M Doc/c-api/type.rst diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index f387279d143ee..7309d7ee2cd39 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -117,6 +117,13 @@ Type Objects If no module is associated with the given type, sets :py:class:`TypeError` and returns ``NULL``. + This function is usually used to get the module in which a method is defined. + Note that in such a method, ``PyType_GetModule(Py_TYPE(self))`` + may not return the intended result. + ``Py_TYPE(self)`` may be a *subclass* of the intended class, and subclasses + are not necessarily defined in the same module as their superclass. + See :c:type:`PyCMethod` to get the class that defines the method. + .. versionadded:: 3.9 .. c:function:: void* PyType_GetModuleState(PyTypeObject *type) @@ -151,9 +158,12 @@ The following functions and structs are used to create If *bases* is ``NULL``, the *Py_tp_base* slot is used instead. If that also is ``NULL``, the new type derives from :class:`object`. - The *module* must be a module object or ``NULL``. + The *module* argument can be used to record the module in which the new + class is defined. It must be a module object or ``NULL``. If not ``NULL``, the module is associated with the new type and can later be retreived with :c:func:`PyType_GetModule`. + The associated module is not inherited by subclasses; it must be specified + for each class individually. This function calls :c:func:`PyType_Ready` on the new type. From webhook-mailer at python.org Fri Aug 28 11:04:12 2020 From: webhook-mailer at python.org (Karthikeyan Singaravelan) Date: Fri, 28 Aug 2020 15:04:12 -0000 Subject: [Python-checkins] [3.8] bpo-41624: fix documentation of typing.Coroutine (GH-21952). (#21983) Message-ID: https://github.com/python/cpython/commit/838316db08a8e3174e4cf8db233ff69d388b3f5c commit: 838316db08a8e3174e4cf8db233ff69d388b3f5c branch: 3.8 author: Karthikeyan Singaravelan committer: GitHub date: 2020-08-28T08:03:40-07:00 summary: [3.8] bpo-41624: fix documentation of typing.Coroutine (GH-21952). (#21983) (cherry picked from commit 8c58d2a216ca2b5965361df9b8d8944bc7d4854d) Co-authored-by: MingZhe Hu Co-authored-by: MingZhe Hu files: A Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 405562ba2a8f8..0706bc870ea3d 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -733,7 +733,7 @@ The module defines the following classes, functions and decorators: .. versionadded:: 3.5.2 -.. class:: Coroutine(Awaitable[V_co], Generic[T_co T_contra, V_co]) +.. class:: Coroutine(Awaitable[V_co], Generic[T_co, T_contra, V_co]) A generic version of :class:`collections.abc.Coroutine`. The variance and order of type variables diff --git a/Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst b/Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst new file mode 100644 index 0000000000000..bdbc5a445f339 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-08-25-15-11-23.bpo-41624.ddjJlN.rst @@ -0,0 +1 @@ +Fix the signature of :class:`typing.Coroutine`. From webhook-mailer at python.org Fri Aug 28 14:48:21 2020 From: webhook-mailer at python.org (SarahPythonista) Date: Fri, 28 Aug 2020 18:48:21 -0000 Subject: [Python-checkins] Fix error in argparse documentation example (GH-17399) Message-ID: https://github.com/python/cpython/commit/8784d3300ec4ffc58bc0e9ab3cff9a24187dbe4c commit: 8784d3300ec4ffc58bc0e9ab3cff9a24187dbe4c branch: master author: SarahPythonista <4283226+SarahPythonista at users.noreply.github.com> committer: GitHub date: 2020-08-28T11:47:58-07:00 summary: Fix error in argparse documentation example (GH-17399) Automerge-Triggered-By: @rhettinger files: M Doc/library/argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 0b64dfe47f768..7a7a4cf94979a 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1162,8 +1162,8 @@ keyword argument to :meth:`~ArgumentParser.add_argument`:: >>> parser.parse_args(['--foo', 'BAR']) Namespace(foo='BAR') >>> parser.parse_args([]) - usage: argparse.py [-h] [--foo FOO] - argparse.py: error: option --foo is required + usage: [-h] --foo FOO + : error: the following arguments are required: --foo As the example shows, if an option is marked as ``required``, :meth:`~ArgumentParser.parse_args` will report an error if that option is not From webhook-mailer at python.org Sat Aug 29 11:00:17 2020 From: webhook-mailer at python.org (Stefan Krah) Date: Sat, 29 Aug 2020 15:00:17 -0000 Subject: [Python-checkins] bpo-19521: Fix parallel build race condition on AIX (GH-21997) Message-ID: https://github.com/python/cpython/commit/e6dcd371b2c54a94584dd124e8c592a496d46a47 commit: e6dcd371b2c54a94584dd124e8c592a496d46a47 branch: master author: Stefan Krah committer: GitHub date: 2020-08-29T17:00:08+02:00 summary: bpo-19521: Fix parallel build race condition on AIX (GH-21997) Patch by Michael Haubenwallner. files: M Makefile.pre.in M configure M configure.ac diff --git a/Makefile.pre.in b/Makefile.pre.in index 0a19313ea9939..5d3ac705a3625 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -161,6 +161,10 @@ BLDSHARED= @BLDSHARED@ $(PY_CORE_LDFLAGS) LDCXXSHARED= @LDCXXSHARED@ DESTSHARED= $(BINLIBDEST)/lib-dynload +# List of exported symbols for AIX +EXPORTSYMS= @EXPORTSYMS@ +EXPORTSFROM= @EXPORTSFROM@ + # Executable suffix (.exe on Windows and Mac OS X) EXE= @EXEEXT@ BUILDEXE= @BUILDEXEEXT@ @@ -571,7 +575,7 @@ clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c $(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --srcdir $(srcdir) # Build the interpreter -$(BUILDPYTHON): Programs/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) +$(BUILDPYTHON): Programs/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS) $(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) platform: $(BUILDPYTHON) pybuilddir.txt @@ -643,6 +647,10 @@ libpython$(LDVERSION).dylib: $(LIBRARY_OBJS) libpython$(VERSION).sl: $(LIBRARY_OBJS) $(LDSHARED) -o $@ $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) +# List of exported symbols for AIX +Modules/python.exp: $(LIBRARY) + $(srcdir)/Modules/makexp_aix $@ "$(EXPORTSFROM)" $? + # Copy up the gdb python hooks into a position where they can be automatically # loaded by gdb during Lib/test/test_gdb.py # @@ -702,7 +710,7 @@ Makefile Modules/config.c: Makefile.pre \ @echo "The Makefile was updated, you may need to re-run make." -Programs/_testembed: Programs/_testembed.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) +Programs/_testembed: Programs/_testembed.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS) $(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/_testembed.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) ############################################################################ diff --git a/configure b/configure index 69e975a4e8ef3..4c18ae7e364ca 100755 --- a/configure +++ b/configure @@ -700,6 +700,8 @@ ARFLAGS ac_ct_AR AR GNULD +EXPORTSFROM +EXPORTSYMS LINKCC LDVERSION RUNSHARED @@ -5799,8 +5801,6 @@ LDVERSION="$VERSION" # If CXX is set, and if it is needed to link a main function that was # compiled with CXX, LINKCC is CXX instead. Always using CXX is undesirable: # python might then depend on the C++ runtime -# This is altered for AIX in order to build the export list before -# linking. { $as_echo "$as_me:${as_lineno-$LINENO}: checking LINKCC" >&5 $as_echo_n "checking LINKCC... " >&6; } @@ -5808,13 +5808,6 @@ if test -z "$LINKCC" then LINKCC='$(PURIFY) $(MAINCC)' case $ac_sys_system in - AIX*) - exp_extra="\"\"" - if test $ac_sys_release -ge 5 -o \ - $ac_sys_release -eq 4 -a `uname -r` -ge 2 ; then - exp_extra="." - fi - LINKCC="\$(srcdir)/Modules/makexp_aix Modules/python.exp $exp_extra \$(LIBRARY); $LINKCC";; QNX*) # qcc must be used because the other compilers do not # support -N. @@ -5824,6 +5817,26 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LINKCC" >&5 $as_echo "$LINKCC" >&6; } +# EXPORTSYMS holds the list of exported symbols for AIX. +# EXPORTSFROM holds the module name exporting symbols on AIX. +EXPORTSYMS= +EXPORTSFROM= + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking EXPORTSYMS" >&5 +$as_echo_n "checking EXPORTSYMS... " >&6; } +case $ac_sys_system in +AIX*) + EXPORTSYMS="Modules/python.exp" + if test $ac_sys_release -ge 5 -o \ + $ac_sys_release -eq 4 -a `uname -r` -ge 2 ; then + EXPORTSFROM=. # the main executable + fi + ;; +esac +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $EXPORTSYMS" >&5 +$as_echo "$EXPORTSYMS" >&6; } + # GNULD is set to "yes" if the GNU linker is used. If this goes wrong # make sure we default having it set to "no": this is used by # distutils.unixccompiler to know if it should add --enable-new-dtags diff --git a/configure.ac b/configure.ac index 07c0585432b06..3b40f39124dad 100644 --- a/configure.ac +++ b/configure.ac @@ -1024,21 +1024,12 @@ LDVERSION="$VERSION" # If CXX is set, and if it is needed to link a main function that was # compiled with CXX, LINKCC is CXX instead. Always using CXX is undesirable: # python might then depend on the C++ runtime -# This is altered for AIX in order to build the export list before -# linking. AC_SUBST(LINKCC) AC_MSG_CHECKING(LINKCC) if test -z "$LINKCC" then LINKCC='$(PURIFY) $(MAINCC)' case $ac_sys_system in - AIX*) - exp_extra="\"\"" - if test $ac_sys_release -ge 5 -o \ - $ac_sys_release -eq 4 -a `uname -r` -ge 2 ; then - exp_extra="." - fi - LINKCC="\$(srcdir)/Modules/makexp_aix Modules/python.exp $exp_extra \$(LIBRARY); $LINKCC";; QNX*) # qcc must be used because the other compilers do not # support -N. @@ -1047,6 +1038,24 @@ then fi AC_MSG_RESULT($LINKCC) +# EXPORTSYMS holds the list of exported symbols for AIX. +# EXPORTSFROM holds the module name exporting symbols on AIX. +EXPORTSYMS= +EXPORTSFROM= +AC_SUBST(EXPORTSYMS) +AC_SUBST(EXPORTSFROM) +AC_MSG_CHECKING(EXPORTSYMS) +case $ac_sys_system in +AIX*) + EXPORTSYMS="Modules/python.exp" + if test $ac_sys_release -ge 5 -o \ + $ac_sys_release -eq 4 -a `uname -r` -ge 2 ; then + EXPORTSFROM=. # the main executable + fi + ;; +esac +AC_MSG_RESULT($EXPORTSYMS) + # GNULD is set to "yes" if the GNU linker is used. If this goes wrong # make sure we default having it set to "no": this is used by # distutils.unixccompiler to know if it should add --enable-new-dtags From webhook-mailer at python.org Sat Aug 29 12:11:08 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Sat, 29 Aug 2020 16:11:08 -0000 Subject: [Python-checkins] bpo-41513: Save unnecessary steps in the hypot() calculation (#21994) Message-ID: https://github.com/python/cpython/commit/27de28607a248e5ffb8838162fca466a58c2e284 commit: 27de28607a248e5ffb8838162fca466a58c2e284 branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-29T09:11:04-07:00 summary: bpo-41513: Save unnecessary steps in the hypot() calculation (#21994) files: M Modules/mathmodule.c diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 1704d8efd31c3..4ff2a069a76c7 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2447,6 +2447,14 @@ precision. When a value at or below 1.0 is correctly rounded, it never goes above 1.0. And when values at or below 1.0 are squared, they remain at or below 1.0, thus preserving the summation invariant. +Another interesting assertion is that csum+lo*lo == csum. In the loop, +each scaled vector element has a magnitude less than 1.0. After the +Veltkamp split, *lo* has a maximum value of 2**-27. So the maximum +value of *lo* squared is 2**-54. The value of ulp(1.0)/2.0 is 2**-53. +Given that csum >= 1.0, we have: + lo**2 <= 2**-54 < 2**-53 == 1/2*ulp(1.0) <= ulp(csum)/2 +Since lo**2 is less than 1/2 ulp(csum), we have csum+lo*lo == csum. + The square root differential correction is needed because a correctly rounded square root of a correctly rounded sum of squares can still be off by as much as one ulp. @@ -2519,11 +2527,8 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) csum += x; frac += (oldcsum - csum) + x; - x = lo * lo; - assert(fabs(csum) >= fabs(x)); - oldcsum = csum; - csum += x; - frac += (oldcsum - csum) + x; + assert(csum + lo * lo == csum); + frac += lo * lo; } h = sqrt(csum - 1.0 + frac); From webhook-mailer at python.org Sat Aug 29 18:33:51 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sat, 29 Aug 2020 22:33:51 -0000 Subject: [Python-checkins] bpo-41634: Fix a typo in the curses documentation (GH-21958) Message-ID: https://github.com/python/cpython/commit/a1473d2c9106abbdc619bdcc973c15a87e3f0f12 commit: a1473d2c9106abbdc619bdcc973c15a87e3f0f12 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-29T18:33:26-04:00 summary: bpo-41634: Fix a typo in the curses documentation (GH-21958) (cherry picked from commit 398575c210f79627830c5c470184f54ace950ac6) Co-authored-by: Zackery Spytz files: M Doc/library/curses.rst diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 957c6f60778fe..7cd20253aeea6 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -682,7 +682,7 @@ the following methods and attributes: window.addch(y, x, ch[, attr]) Paint character *ch* at ``(y, x)`` with attributes *attr*, overwriting any - character previously painter at that location. By default, the character + character previously painted at that location. By default, the character position and attributes are the current settings for the window object. .. note:: From webhook-mailer at python.org Sat Aug 29 19:47:42 2020 From: webhook-mailer at python.org (Roger Iyengar) Date: Sat, 29 Aug 2020 23:47:42 -0000 Subject: [Python-checkins] Improve asyncio-dev 'Concurrency and Multithreading' docs (GH-20882) Message-ID: https://github.com/python/cpython/commit/c68c5af2dc5ada8875a662f2beaac6234eae2a5a commit: c68c5af2dc5ada8875a662f2beaac6234eae2a5a branch: master author: Roger Iyengar committer: GitHub date: 2020-08-29T16:47:38-07:00 summary: Improve asyncio-dev 'Concurrency and Multithreading' docs (GH-20882) I added some information to the `Concurrency and Multithreading` section of the `Developing with asyncio` guide. This is all information that would have helped me when I started using asyncio. I incorrectly assumed that `loop.call_soon_threadsafe()` and `run_coroutine_threadsafe()` could be called from a thread in a process separate from the one that the event loop is running in. Explicitly stating that this will not work will probably help some people starting out with asyncio in the future. I also added references to some other functions that can be used for inter-process communication without blocking the event loop. The section already mentions running blocking code in a ThreadPoolExecutor, but I think listing these other options in this section will also be helpful. files: M Doc/library/asyncio-dev.rst diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 77f8067197836..02a00033152ab 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -107,6 +107,16 @@ The :meth:`loop.run_in_executor` method can be used with a blocking code in a different OS thread without blocking the OS thread that the event loop runs in. +There is currently no way to schedule coroutines or callbacks directly +from a different process (such as one started with +:mod:`multiprocessing`). The :ref:`Event Loop Methods ` +section lists APIs that can read from pipes and watch file descriptors +without blocking the event loop. In addition, asyncio's +:ref:`Subprocess ` APIs provide a way to start a +process and communicate with it from the event loop. Lastly, the +aforementioned :meth:`loop.run_in_executor` method can also be used +with a :class:`concurrent.futures.ProcessPoolExecutor` to execute +code in a different process. .. _asyncio-handle-blocking: From webhook-mailer at python.org Sun Aug 30 00:53:17 2020 From: webhook-mailer at python.org (Tony Solomonik) Date: Sun, 30 Aug 2020 04:53:17 -0000 Subject: [Python-checkins] closes bpo-41533: Fix a potential memory leak when allocating a stack (GH-21847) Message-ID: https://github.com/python/cpython/commit/75c80b0bda89debf312f075716b8c467d411f90e commit: 75c80b0bda89debf312f075716b8c467d411f90e branch: master author: Tony Solomonik committer: GitHub date: 2020-08-29T23:53:08-05:00 summary: closes bpo-41533: Fix a potential memory leak when allocating a stack (GH-21847) Free the stack allocated in va_build_stack if do_mkstack fails and the stack is not a small_stack files: A Misc/NEWS.d/next/Core and Builtins/2020-08-12-20-29-57.bpo-41533.4pcVAc.rst M Python/modsupport.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-12-20-29-57.bpo-41533.4pcVAc.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-12-20-29-57.bpo-41533.4pcVAc.rst new file mode 100644 index 0000000000000..e166f0c0b621a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-12-20-29-57.bpo-41533.4pcVAc.rst @@ -0,0 +1,2 @@ +Free the stack allocated in ``va_build_stack`` if ``do_mkstack`` fails and +the stack is not a ``small_stack``. diff --git a/Python/modsupport.c b/Python/modsupport.c index 2637039d4a151..2dabcf383409e 100644 --- a/Python/modsupport.c +++ b/Python/modsupport.c @@ -622,6 +622,9 @@ va_build_stack(PyObject **small_stack, Py_ssize_t small_stack_len, va_end(lva); if (res < 0) { + if (stack != small_stack) { + PyMem_Free(stack); + } return NULL; } From webhook-mailer at python.org Sun Aug 30 03:20:44 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 30 Aug 2020 07:20:44 -0000 Subject: [Python-checkins] bpo-41524: fix pointer bug in PyOS_mystr{n}icmp (GH-21845) (GH-22016) Message-ID: https://github.com/python/cpython/commit/85ca9c049c5a490d143d28933bbb02ab80394ed8 commit: 85ca9c049c5a490d143d28933bbb02ab80394ed8 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-30T16:20:40+09:00 summary: bpo-41524: fix pointer bug in PyOS_mystr{n}icmp (GH-21845) (GH-22016) files: A Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst M Python/pystrcmp.c diff --git a/Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst b/Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst new file mode 100644 index 0000000000000..4704e29be29bb --- /dev/null +++ b/Misc/NEWS.d/next/C API/2020-08-12-17-09-06.bpo-41524.u6Xfr2.rst @@ -0,0 +1,2 @@ +Fix bug in PyOS_mystrnicmp and PyOS_mystricmp that incremented +pointers beyond the end of a string. \ No newline at end of file diff --git a/Python/pystrcmp.c b/Python/pystrcmp.c index f9c2277cb56dc..9224ce4c70605 100644 --- a/Python/pystrcmp.c +++ b/Python/pystrcmp.c @@ -6,21 +6,25 @@ int PyOS_mystrnicmp(const char *s1, const char *s2, Py_ssize_t size) { + const unsigned char *p1, *p2; if (size == 0) return 0; - while ((--size > 0) && - (tolower((unsigned)*s1) == tolower((unsigned)*s2))) { - if (!*s1++ || !*s2++) - break; + p1 = (const unsigned char *)s1; + p2 = (const unsigned char *)s2; + for (; (--size > 0) && *p1 && *p2 && (tolower(*p1) == tolower(*p2)); + p1++, p2++) { + ; } - return tolower((unsigned)*s1) - tolower((unsigned)*s2); + return tolower(*p1) - tolower(*p2); } int PyOS_mystricmp(const char *s1, const char *s2) { - while (*s1 && (tolower((unsigned)*s1++) == tolower((unsigned)*s2++))) { + const unsigned char *p1 = (const unsigned char *)s1; + const unsigned char *p2 = (const unsigned char *)s2; + for (; *p1 && *p2 && (tolower(*p1) == tolower(*p2)); p1++, p2++) { ; } - return (tolower((unsigned)*s1) - tolower((unsigned)*s2)); + return (tolower(*p1) - tolower(*p2)); } From webhook-mailer at python.org Sun Aug 30 13:00:19 2020 From: webhook-mailer at python.org (Raymond Hettinger) Date: Sun, 30 Aug 2020 17:00:19 -0000 Subject: [Python-checkins] Further improve accuracy of math.hypot() (GH-22013) Message-ID: https://github.com/python/cpython/commit/92c38164a42572e2bc0b1b1490bec2369480ae08 commit: 92c38164a42572e2bc0b1b1490bec2369480ae08 branch: master author: Raymond Hettinger committer: GitHub date: 2020-08-30T10:00:11-07:00 summary: Further improve accuracy of math.hypot() (GH-22013) files: M Modules/mathmodule.c diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 4ff2a069a76c7..6621951ee97d2 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2455,6 +2455,9 @@ Given that csum >= 1.0, we have: lo**2 <= 2**-54 < 2**-53 == 1/2*ulp(1.0) <= ulp(csum)/2 Since lo**2 is less than 1/2 ulp(csum), we have csum+lo*lo == csum. +To minimize loss of information during the accumulation of fractional +values, the lo**2 term has a separate accumulator. + The square root differential correction is needed because a correctly rounded square root of a correctly rounded sum of squares can still be off by as much as one ulp. @@ -2475,7 +2478,8 @@ Borges' ALGORITHM 4 and 5. 1. Veltkamp-Dekker splitting: http://csclub.uwaterloo.ca/~pbarfuss/dekker1971.pdf 2. Compensated summation: http://www.ti3.tu-harburg.de/paper/rump/Ru08b.pdf -3. Square root diffential correction: https://arxiv.org/pdf/1904.09481.pdf +3. Square root differential correction: https://arxiv.org/pdf/1904.09481.pdf +4. https://www.wolframalpha.com/input/?i=Maclaurin+series+sqrt%28h**2+%2B+x%29+at+x%3D0 */ @@ -2483,7 +2487,7 @@ static inline double vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) { const double T27 = 134217729.0; /* ldexp(1.0, 27)+1.0) */ - double x, csum = 1.0, oldcsum, frac = 0.0, scale; + double x, csum = 1.0, oldcsum, frac = 0.0, frac_lo = 0.0, scale; double t, hi, lo, h; int max_e; Py_ssize_t i; @@ -2528,8 +2532,9 @@ vector_norm(Py_ssize_t n, double *vec, double max, int found_nan) frac += (oldcsum - csum) + x; assert(csum + lo * lo == csum); - frac += lo * lo; + frac_lo += lo * lo; } + frac += frac_lo; h = sqrt(csum - 1.0 + frac); x = h; From webhook-mailer at python.org Sun Aug 30 13:29:58 2020 From: webhook-mailer at python.org (Irit Katriel) Date: Sun, 30 Aug 2020 17:29:58 -0000 Subject: [Python-checkins] bpo-39994: Fix pprint handling of dict subclasses that override __repr__ (GH-21892) Message-ID: https://github.com/python/cpython/commit/582f13786bb75c73d609790967fea03a5b50148a commit: 582f13786bb75c73d609790967fea03a5b50148a branch: master author: Irit Katriel committer: GitHub date: 2020-08-30T20:29:53+03:00 summary: bpo-39994: Fix pprint handling of dict subclasses that override __repr__ (GH-21892) Co-authored-by: Palak Kumar Jha files: A Misc/NEWS.d/next/Library/2020-08-15-18-17-21.bpo-39994.dOgPOh.rst M Lib/pprint.py M Lib/test/test_pprint.py M Misc/ACKS diff --git a/Lib/pprint.py b/Lib/pprint.py index 7c1118a484b26..213998e3491ef 100644 --- a/Lib/pprint.py +++ b/Lib/pprint.py @@ -176,12 +176,6 @@ def _format(self, object, stream, indent, allowance, context, level): p(self, object, stream, indent, allowance, context, level + 1) del context[objid] return - elif isinstance(object, dict): - context[objid] = 1 - self._pprint_dict(object, stream, indent, allowance, - context, level + 1) - del context[objid] - return stream.write(rep) _dispatch = {} diff --git a/Lib/test/test_pprint.py b/Lib/test/test_pprint.py index cf3e4f093b16e..8ee18e8fef84f 100644 --- a/Lib/test/test_pprint.py +++ b/Lib/test/test_pprint.py @@ -18,6 +18,10 @@ class list3(list): def __repr__(self): return list.__repr__(self) +class list_custom_repr(list): + def __repr__(self): + return '*'*len(list.__repr__(self)) + class tuple2(tuple): pass @@ -25,6 +29,10 @@ class tuple3(tuple): def __repr__(self): return tuple.__repr__(self) +class tuple_custom_repr(tuple): + def __repr__(self): + return '*'*len(tuple.__repr__(self)) + class set2(set): pass @@ -32,6 +40,10 @@ class set3(set): def __repr__(self): return set.__repr__(self) +class set_custom_repr(set): + def __repr__(self): + return '*'*len(set.__repr__(self)) + class frozenset2(frozenset): pass @@ -39,6 +51,10 @@ class frozenset3(frozenset): def __repr__(self): return frozenset.__repr__(self) +class frozenset_custom_repr(frozenset): + def __repr__(self): + return '*'*len(frozenset.__repr__(self)) + class dict2(dict): pass @@ -46,6 +62,10 @@ class dict3(dict): def __repr__(self): return dict.__repr__(self) +class dict_custom_repr(dict): + def __repr__(self): + return '*'*len(dict.__repr__(self)) + class Unorderable: def __repr__(self): return str(id(self)) @@ -155,7 +175,8 @@ def test_unreadable(self): "expected not isreadable for %r" % (unreadable,)) def test_same_as_repr(self): - # Simple objects, small containers and classes that overwrite __repr__ + # Simple objects, small containers and classes that override __repr__ + # to directly call super's __repr__. # For those the result should be the same as repr(). # Ahem. The docs don't say anything about that -- this appears to # be testing an implementation quirk. Starting in Python 2.5, it's @@ -187,6 +208,32 @@ def test_same_as_repr(self): .replace('\n', ' '), native) self.assertEqual(pprint.saferepr(simple), native) + def test_container_repr_override_called(self): + N = 1000 + # Ensure that __repr__ override is called for subclasses of containers + + for cont in (list_custom_repr(), + list_custom_repr([1,2,3]), + list_custom_repr(range(N)), + tuple_custom_repr(), + tuple_custom_repr([1,2,3]), + tuple_custom_repr(range(N)), + set_custom_repr(), + set_custom_repr([1,2,3]), + set_custom_repr(range(N)), + frozenset_custom_repr(), + frozenset_custom_repr([1,2,3]), + frozenset_custom_repr(range(N)), + dict_custom_repr(), + dict_custom_repr({5: 6}), + dict_custom_repr(zip(range(N),range(N))), + ): + native = repr(cont) + expected = '*' * len(native) + self.assertEqual(pprint.pformat(cont), expected) + self.assertEqual(pprint.pformat(cont, width=1, indent=0), expected) + self.assertEqual(pprint.saferepr(cont), expected) + def test_basic_line_wrap(self): # verify basic line-wrapping operation o = {'RPM_cal': 0, diff --git a/Misc/ACKS b/Misc/ACKS index 1599b09c692b7..a2cdeb8504059 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -856,6 +856,7 @@ Per ?yvind Karlsen Anton Kasyanov Lou Kates Makoto Kato +Irit Katriel Hiroaki Kawai Dmitry Kazakov Brian Kearns diff --git a/Misc/NEWS.d/next/Library/2020-08-15-18-17-21.bpo-39994.dOgPOh.rst b/Misc/NEWS.d/next/Library/2020-08-15-18-17-21.bpo-39994.dOgPOh.rst new file mode 100644 index 0000000000000..46876c15ea199 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-15-18-17-21.bpo-39994.dOgPOh.rst @@ -0,0 +1 @@ +Fixed pprint's handling of dict subclasses that override __repr__. From webhook-mailer at python.org Sun Aug 30 15:03:19 2020 From: webhook-mailer at python.org (Vinay Sharma) Date: Sun, 30 Aug 2020 19:03:19 -0000 Subject: [Python-checkins] bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556) Message-ID: https://github.com/python/cpython/commit/475a5fbb5644ea200c990d85d8c264e78ab6c7ea commit: 475a5fbb5644ea200c990d85d8c264e78ab6c7ea branch: master author: Vinay Sharma committer: GitHub date: 2020-08-30T20:03:11+01:00 summary: bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556) files: A Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst M Lib/multiprocessing/shared_memory.py M Lib/test/_test_multiprocessing.py diff --git a/Lib/multiprocessing/shared_memory.py b/Lib/multiprocessing/shared_memory.py index a3a5fcf4aba1e..122b3fcebf3fe 100644 --- a/Lib/multiprocessing/shared_memory.py +++ b/Lib/multiprocessing/shared_memory.py @@ -76,6 +76,8 @@ def __init__(self, name=None, create=False, size=0): raise ValueError("'size' must be a positive integer") if create: self._flags = _O_CREX | os.O_RDWR + if size == 0: + raise ValueError("'size' must be a positive number different from zero") if name is None and not self._flags & os.O_EXCL: raise ValueError("'name' can only be None if create=True") diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 6a0e1016aff8d..fd3b4303f034c 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3880,6 +3880,18 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory): sms.close() + # Test creating a shared memory segment with negative size + with self.assertRaises(ValueError): + sms_invalid = shared_memory.SharedMemory(create=True, size=-1) + + # Test creating a shared memory segment with size 0 + with self.assertRaises(ValueError): + sms_invalid = shared_memory.SharedMemory(create=True, size=0) + + # Test creating a shared memory segment without size argument + with self.assertRaises(ValueError): + sms_invalid = shared_memory.SharedMemory(create=True) + def test_shared_memory_across_processes(self): # bpo-40135: don't define shared memory block's name in case of # the failure when we run multiprocessing tests in parallel. diff --git a/Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst b/Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst new file mode 100644 index 0000000000000..475bc9bddb0d5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst @@ -0,0 +1 @@ +Prevent creating :class:`shared_memory.SharedMemory` objects with :code:`size=0`. From webhook-mailer at python.org Sun Aug 30 15:42:31 2020 From: webhook-mailer at python.org (Miss Islington (bot)) Date: Sun, 30 Aug 2020 19:42:31 -0000 Subject: [Python-checkins] bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556) (GH-22019) Message-ID: https://github.com/python/cpython/commit/38e32872eb3cf0dc9dd8cef9b05e47ba03638d34 commit: 38e32872eb3cf0dc9dd8cef9b05e47ba03638d34 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: GitHub date: 2020-08-30T20:42:27+01:00 summary: bpo-41344: Raise ValueError when creating shared memory of size 0 (GH-21556) (GH-22019) (cherry picked from commit 475a5fbb5644ea200c990d85d8c264e78ab6c7ea) Co-authored-by: Vinay Sharma Co-authored-by: Vinay Sharma files: A Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst M Lib/multiprocessing/shared_memory.py M Lib/test/_test_multiprocessing.py diff --git a/Lib/multiprocessing/shared_memory.py b/Lib/multiprocessing/shared_memory.py index f92eb012c8316..e5b5e6c098a02 100644 --- a/Lib/multiprocessing/shared_memory.py +++ b/Lib/multiprocessing/shared_memory.py @@ -75,6 +75,8 @@ def __init__(self, name=None, create=False, size=0): raise ValueError("'size' must be a positive integer") if create: self._flags = _O_CREX | os.O_RDWR + if size == 0: + raise ValueError("'size' must be a positive number different from zero") if name is None and not self._flags & os.O_EXCL: raise ValueError("'name' can only be None if create=True") diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 8626aa37c186e..ee9b47bb5f153 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3827,6 +3827,18 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory): sms.close() + # Test creating a shared memory segment with negative size + with self.assertRaises(ValueError): + sms_invalid = shared_memory.SharedMemory(create=True, size=-1) + + # Test creating a shared memory segment with size 0 + with self.assertRaises(ValueError): + sms_invalid = shared_memory.SharedMemory(create=True, size=0) + + # Test creating a shared memory segment without size argument + with self.assertRaises(ValueError): + sms_invalid = shared_memory.SharedMemory(create=True) + def test_shared_memory_across_processes(self): # bpo-40135: don't define shared memory block's name in case of # the failure when we run multiprocessing tests in parallel. diff --git a/Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst b/Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst new file mode 100644 index 0000000000000..475bc9bddb0d5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-07-20-13-27-48.bpo-41344.iKipNd.rst @@ -0,0 +1 @@ +Prevent creating :class:`shared_memory.SharedMemory` objects with :code:`size=0`. From webhook-mailer at python.org Sun Aug 30 16:37:02 2020 From: webhook-mailer at python.org (Andre Delfino) Date: Sun, 30 Aug 2020 20:37:02 -0000 Subject: [Python-checkins] [doc] Fix markup in logging (GH-22008) Message-ID: https://github.com/python/cpython/commit/c3a651ad2544d7d1be389b63e9a4a58a92a31623 commit: c3a651ad2544d7d1be389b63e9a4a58a92a31623 branch: master author: Andre Delfino committer: GitHub date: 2020-08-30T21:36:58+01:00 summary: [doc] Fix markup in logging (GH-22008) files: M Doc/library/logging.rst diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 3ff67f76cc3c5..a446c80ece604 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -1222,7 +1222,7 @@ functions. | | opening the output file. If not specified, | | | the value 'backslashreplace' is used. Note | | | that if ``None`` is specified, it will be | - | | passed as such to func:`open`, which means | + | | passed as such to :func:`open`, which means | | | that it will be treated the same as passing | | | 'errors'. | +--------------+---------------------------------------------+ From webhook-mailer at python.org Mon Aug 31 15:58:02 2020 From: webhook-mailer at python.org (Ben Darnell) Date: Mon, 31 Aug 2020 19:58:02 -0000 Subject: [Python-checkins] bpo-39010: Fix errors logged on proactor loop restart (#22017) Message-ID: https://github.com/python/cpython/commit/ea5a6363c3f8cc90b7c0cc573922b10f296073b6 commit: ea5a6363c3f8cc90b7c0cc573922b10f296073b6 branch: master author: Ben Darnell committer: GitHub date: 2020-08-31T12:57:52-07:00 summary: bpo-39010: Fix errors logged on proactor loop restart (#22017) Stopping and restarting a proactor event loop on windows can lead to spurious errors logged (ConnectionResetError while reading from the self pipe). This fixes the issue by ensuring that we don't attempt to start multiple copies of the self-pipe reading loop. files: A Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst M Lib/asyncio/proactor_events.py M Lib/asyncio/windows_events.py M Lib/test/test_asyncio/test_proactor_events.py M Lib/test/test_asyncio/test_windows_events.py diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py index d0b7100f5e056..4670bd683ab32 100644 --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -768,6 +768,14 @@ def _loop_self_reading(self, f=None): try: if f is not None: f.result() # may raise + if self._self_reading_future is not f: + # When we scheduled this Future, we assigned it to + # _self_reading_future. If it's not there now, something has + # tried to cancel the loop while this callback was still in the + # queue (see windows_events.ProactorEventLoop.run_forever). In + # that case stop here instead of continuing to schedule a new + # iteration. + return f = self._proactor.recv(self._ssock, 4096) except exceptions.CancelledError: # _close_self_pipe() has been called, stop waiting for data diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py index a6759b78bd801..5e7cd795895d6 100644 --- a/Lib/asyncio/windows_events.py +++ b/Lib/asyncio/windows_events.py @@ -318,8 +318,12 @@ def run_forever(self): if self._self_reading_future is not None: ov = self._self_reading_future._ov self._self_reading_future.cancel() - # self_reading_future was just cancelled so it will never be signalled - # Unregister it otherwise IocpProactor.close will wait for it forever + # self_reading_future was just cancelled so if it hasn't been + # finished yet, it never will be (it's possible that it has + # already finished and its callback is waiting in the queue, + # where it could still happen if the event loop is restarted). + # Unregister it otherwise IocpProactor.close will wait for it + # forever if ov is not None: self._proactor._unregister(ov) self._self_reading_future = None diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py index d0ab38743ecd1..4c8906d531ce5 100644 --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -753,6 +753,7 @@ def test_loop_self_reading(self): def test_loop_self_reading_fut(self): fut = mock.Mock() + self.loop._self_reading_future = fut self.loop._loop_self_reading(fut) self.assertTrue(fut.result.called) self.proactor.recv.assert_called_with(self.ssock, 4096) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py index 6b005702c9be7..33388a87d48f3 100644 --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -211,6 +211,26 @@ def test_wait_for_handle_cancel(self): fut.cancel() fut.cancel() + def test_read_self_pipe_restart(self): + # Regression test for https://bugs.python.org/issue39010 + # Previously, restarting a proactor event loop in certain states + # would lead to spurious ConnectionResetErrors being logged. + self.loop.call_exception_handler = mock.Mock() + # Start an operation in another thread so that the self-pipe is used. + # This is theoretically timing-dependent (the task in the executor + # must complete before our start/stop cycles), but in practice it + # seems to work every time. + f = self.loop.run_in_executor(None, lambda: None) + self.loop.stop() + self.loop.run_forever() + self.loop.stop() + self.loop.run_forever() + # If we don't wait for f to complete here, we may get another + # warning logged about a thread that didn't shut down cleanly. + self.loop.run_until_complete(f) + self.loop.close() + self.assertFalse(self.loop.call_exception_handler.called) + class WinPolicyTests(test_utils.TestCase): diff --git a/Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst b/Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst new file mode 100644 index 0000000000000..0d9015b490e4c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-30-10-24-26.bpo-39010._mzXJW.rst @@ -0,0 +1,2 @@ +Restarting a ``ProactorEventLoop`` on Windows no longer logs spurious +``ConnectionResetErrors``. From webhook-mailer at python.org Mon Aug 10 10:48:29 2020 From: webhook-mailer at python.org (Filipe =?utf-8?q?La=C3=ADns?=) Date: Mon, 10 Aug 2020 14:48:29 -0000 Subject: [Python-checkins] (no subject) Message-ID: To: python-checkins at python.org Subject: bpo-16995: add support for base32 extended hex (base32hex) (GH-20441) Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 https://github.com/python/cpython/commit/4ce6faa6c9591de6079347eccc9e61ae4e8d= 9e31 commit: 4ce6faa6c9591de6079347eccc9e61ae4e8d9e31 branch: master author: Filipe La=C3=ADns committer: GitHub date: 2020-08-10T07:48:20-07:00 summary: bpo-16995: add support for base32 extended hex (base32hex) (GH-20441) cc @pganssle Automerge-Triggered-By: @pganssle files: A Misc/NEWS.d/next/Library/2020-05-27-00-09-52.bpo-16995.4niOT7.rst M Doc/library/base64.rst M Doc/whatsnew/3.10.rst M Lib/base64.py M Lib/test/test_base64.py diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 1ff22a00d6199..2f24bb63912fb 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -124,7 +124,7 @@ The modern interface provides: whether a lowercase alphabet is acceptable as input. For security purpos= es, the default is ``False``. =20 - :rfc:`3548` allows for optional mapping of the digit 0 (zero) to the lett= er O + :rfc:`4648` allows for optional mapping of the digit 0 (zero) to the lett= er O (oh), and for optional mapping of the digit 1 (one) to either the letter = I (eye) or letter L (el). The optional argument *map01* when not ``None``, speci= fies which letter the digit 1 should be mapped to (when *map01* is not ``None`= `, the @@ -136,6 +136,27 @@ The modern interface provides: input. =20 =20 +.. function:: b32hexencode(s) + + Similar to :func:`b32encode` but uses the Extended Hex Alphabet, as defin= ed in + :rfc:`4648`. + + .. versionadded:: 3.10 + + +.. function:: b32hexdecode(s, casefold=3DFalse) + + Similar to :func:`b32decode` but uses the Extended Hex Alphabet, as defin= ed in + :rfc:`4648`. + + This version does not allow the digit 0 (zero) to the letter O (oh) and d= igit + 1 (one) to either the letter I (eye) or letter L (el) mappings, all these + characters are included in the Extended Hex Alphabet and are not + interchangable. + + .. versionadded:: 3.10 + + .. function:: b16encode(s) =20 Encode the :term:`bytes-like object` *s* using Base16 and return the diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 2af0ea3f4dd64..eb5ae01a7c04d 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -103,6 +103,12 @@ New Modules Improved Modules =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 +base64 +------ + +Add :func:`base64.b32hexencode` and :func:`base64.b32hexdecode` to support t= he +Base32 Encoding with Extended Hex Alphabet. + curses ------ =20 diff --git a/Lib/base64.py b/Lib/base64.py index a28109f8a7f9c..539ad16f0e86d 100755 --- a/Lib/base64.py +++ b/Lib/base64.py @@ -16,7 +16,7 @@ 'encode', 'decode', 'encodebytes', 'decodebytes', # Generalized interface for other encodings 'b64encode', 'b64decode', 'b32encode', 'b32decode', - 'b16encode', 'b16decode', + 'b32hexencode', 'b32hexdecode', 'b16encode', 'b16decode', # Base85 and Ascii85 encodings 'b85encode', 'b85decode', 'a85encode', 'a85decode', # Standard Base64 encoding @@ -135,19 +135,40 @@ def urlsafe_b64decode(s): =20 =20 # Base32 encoding/decoding must be done in Python +_B32_ENCODE_DOCSTRING =3D ''' +Encode the bytes-like objects using {encoding} and return a bytes object. +''' +_B32_DECODE_DOCSTRING =3D ''' +Decode the {encoding} encoded bytes-like object or ASCII string s. + +Optional casefold is a flag specifying whether a lowercase alphabet is +acceptable as input. For security purposes, the default is False. +{extra_args} +The result is returned as a bytes object. A binascii.Error is raised if +the input is incorrectly padded or if there are non-alphabet +characters present in the input. +''' +_B32_DECODE_MAP01_DOCSTRING =3D ''' +RFC 3548 allows for optional mapping of the digit 0 (zero) to the +letter O (oh), and for optional mapping of the digit 1 (one) to +either the letter I (eye) or letter L (el). The optional argument +map01 when not None, specifies which letter the digit 1 should be +mapped to (when map01 is not None, the digit 0 is always mapped to +the letter O). For security purposes the default is None, so that +0 and 1 are not allowed in the input. +''' _b32alphabet =3D b'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567' -_b32tab2 =3D None -_b32rev =3D None +_b32hexalphabet =3D b'0123456789ABCDEFGHIJKLMNOPQRSTUV' +_b32tab2 =3D {} +_b32rev =3D {} =20 -def b32encode(s): - """Encode the bytes-like object s using Base32 and return a bytes object. - """ +def _b32encode(alphabet, s): global _b32tab2 # Delay the initialization of the table to not waste memory # if the function is never called - if _b32tab2 is None: - b32tab =3D [bytes((i,)) for i in _b32alphabet] - _b32tab2 =3D [a + b for a in b32tab for b in b32tab] + if alphabet not in _b32tab2: + b32tab =3D [bytes((i,)) for i in alphabet] + _b32tab2[alphabet] =3D [a + b for a in b32tab for b in b32tab] b32tab =3D None =20 if not isinstance(s, bytes_types): @@ -158,7 +179,7 @@ def b32encode(s): s =3D s + b'\0' * (5 - leftover) # Don't use +=3D ! encoded =3D bytearray() from_bytes =3D int.from_bytes - b32tab2 =3D _b32tab2 + b32tab2 =3D _b32tab2[alphabet] for i in range(0, len(s), 5): c =3D from_bytes(s[i: i + 5], 'big') encoded +=3D (b32tab2[c >> 30] + # bits 1 - 10 @@ -177,29 +198,12 @@ def b32encode(s): encoded[-1:] =3D b'=3D' return bytes(encoded) =20 -def b32decode(s, casefold=3DFalse, map01=3DNone): - """Decode the Base32 encoded bytes-like object or ASCII string s. - - Optional casefold is a flag specifying whether a lowercase alphabet is - acceptable as input. For security purposes, the default is False. - - RFC 3548 allows for optional mapping of the digit 0 (zero) to the - letter O (oh), and for optional mapping of the digit 1 (one) to - either the letter I (eye) or letter L (el). The optional argument - map01 when not None, specifies which letter the digit 1 should be - mapped to (when map01 is not None, the digit 0 is always mapped to - the letter O). For security purposes the default is None, so that - 0 and 1 are not allowed in the input. - - The result is returned as a bytes object. A binascii.Error is raised if - the input is incorrectly padded or if there are non-alphabet - characters present in the input. - """ +def _b32decode(alphabet, s, casefold=3DFalse, map01=3DNone): global _b32rev # Delay the initialization of the table to not waste memory # if the function is never called - if _b32rev is None: - _b32rev =3D {v: k for k, v in enumerate(_b32alphabet)} + if alphabet not in _b32rev: + _b32rev[alphabet] =3D {v: k for k, v in enumerate(alphabet)} s =3D _bytes_from_decode_data(s) if len(s) % 8: raise binascii.Error('Incorrect padding') @@ -220,7 +224,7 @@ def b32decode(s, casefold=3DFalse, map01=3DNone): padchars =3D l - len(s) # Now decode the full quanta decoded =3D bytearray() - b32rev =3D _b32rev + b32rev =3D _b32rev[alphabet] for i in range(0, len(s), 8): quanta =3D s[i: i + 8] acc =3D 0 @@ -241,6 +245,26 @@ def b32decode(s, casefold=3DFalse, map01=3DNone): return bytes(decoded) =20 =20 +def b32encode(s): + return _b32encode(_b32alphabet, s) +b32encode.__doc__ =3D _B32_ENCODE_DOCSTRING.format(encoding=3D'base32') + +def b32decode(s, casefold=3DFalse, map01=3DNone): + return _b32decode(_b32alphabet, s, casefold, map01) +b32decode.__doc__ =3D _B32_DECODE_DOCSTRING.format(encoding=3D'base32', + extra_args=3D_B32_DECODE_MAP01_DOCST= RING) + +def b32hexencode(s): + return _b32encode(_b32hexalphabet, s) +b32hexencode.__doc__ =3D _B32_ENCODE_DOCSTRING.format(encoding=3D'base32hex') + +def b32hexdecode(s, casefold=3DFalse): + # base32hex does not have the 01 mapping + return _b32decode(_b32hexalphabet, s, casefold) +b32hexdecode.__doc__ =3D _B32_DECODE_DOCSTRING.format(encoding=3D'base32hex', + extra_args=3D'') + + # RFC 3548, Base 16 Alphabet specifies uppercase, but hexlify() returns # lowercase. The RFC also recommends against accepting input case # insensitively. diff --git a/Lib/test/test_base64.py b/Lib/test/test_base64.py index 1f67e46cd2267..4f62c4115f60a 100644 --- a/Lib/test/test_base64.py +++ b/Lib/test/test_base64.py @@ -351,6 +351,76 @@ def test_b32decode_error(self): with self.assertRaises(binascii.Error): base64.b32decode(data.decode('ascii')) =20 + def test_b32hexencode(self): + test_cases =3D [ + # to_encode, expected + (b'', b''), + (b'\x00', b'00=3D=3D=3D=3D=3D=3D'), + (b'a', b'C4=3D=3D=3D=3D=3D=3D'), + (b'ab', b'C5H0=3D=3D=3D=3D'), + (b'abc', b'C5H66=3D=3D=3D'), + (b'abcd', b'C5H66P0=3D'), + (b'abcde', b'C5H66P35'), + ] + for to_encode, expected in test_cases: + with self.subTest(to_decode=3Dto_encode): + self.assertEqual(base64.b32hexencode(to_encode), expected) + + def test_b32hexencode_other_types(self): + self.check_other_types(base64.b32hexencode, b'abcd', b'C5H66P0=3D') + self.check_encode_type_errors(base64.b32hexencode) + + def test_b32hexdecode(self): + test_cases =3D [ + # to_decode, expected, casefold + (b'', b'', False), + (b'00=3D=3D=3D=3D=3D=3D', b'\x00', False), + (b'C4=3D=3D=3D=3D=3D=3D', b'a', False), + (b'C5H0=3D=3D=3D=3D', b'ab', False), + (b'C5H66=3D=3D=3D', b'abc', False), + (b'C5H66P0=3D', b'abcd', False), + (b'C5H66P35', b'abcde', False), + (b'', b'', True), + (b'00=3D=3D=3D=3D=3D=3D', b'\x00', True), + (b'C4=3D=3D=3D=3D=3D=3D', b'a', True), + (b'C5H0=3D=3D=3D=3D', b'ab', True), + (b'C5H66=3D=3D=3D', b'abc', True), + (b'C5H66P0=3D', b'abcd', True), + (b'C5H66P35', b'abcde', True), + (b'c4=3D=3D=3D=3D=3D=3D', b'a', True), + (b'c5h0=3D=3D=3D=3D', b'ab', True), + (b'c5h66=3D=3D=3D', b'abc', True), + (b'c5h66p0=3D', b'abcd', True), + (b'c5h66p35', b'abcde', True), + ] + for to_decode, expected, casefold in test_cases: + with self.subTest(to_decode=3Dto_decode, casefold=3Dcasefold): + self.assertEqual(base64.b32hexdecode(to_decode, casefold), + expected) + self.assertEqual(base64.b32hexdecode(to_decode.decode('ascii= '), + casefold), expected) + + def test_b32hexdecode_other_types(self): + self.check_other_types(base64.b32hexdecode, b'C5H66=3D=3D=3D', b'abc= ') + self.check_decode_type_errors(base64.b32hexdecode) + + def test_b32hexdecode_error(self): + tests =3D [b'abc', b'ABCDEF=3D=3D', b'=3D=3DABCDEF', b'c4=3D=3D=3D= =3D=3D=3D'] + prefixes =3D [b'M', b'ME', b'MFRA', b'MFRGG', b'MFRGGZA', b'MFRGGZDF= '] + for i in range(0, 17): + if i: + tests.append(b'=3D'*i) + for prefix in prefixes: + if len(prefix) + i !=3D 8: + tests.append(prefix + b'=3D'*i) + for data in tests: + with self.subTest(to_decode=3Ddata): + with self.assertRaises(binascii.Error): + base64.b32hexdecode(data) + with self.assertRaises(binascii.Error): + base64.b32hexdecode(data.decode('ascii')) + + def test_b16encode(self): eq =3D self.assertEqual eq(base64.b16encode(b'\x01\x02\xab\xcd\xef'), b'0102ABCDEF') diff --git a/Misc/NEWS.d/next/Library/2020-05-27-00-09-52.bpo-16995.4niOT7.rs= t b/Misc/NEWS.d/next/Library/2020-05-27-00-09-52.bpo-16995.4niOT7.rst new file mode 100644 index 0000000000000..88b95998d085f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-05-27-00-09-52.bpo-16995.4niOT7.rst @@ -0,0 +1,2 @@ +Add :func:`base64.b32hexencode` and :func:`base64.b32hexdecode` to support t= he +Base32 Encoding with Extended Hex Alphabet. From webhook-mailer at python.org Wed Aug 26 20:24:47 2020 From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa) Date: Thu, 27 Aug 2020 00:24:47 -0000 Subject: [Python-checkins] (no subject) Message-ID: To: python-checkins at python.org Subject: bpo-33660: Fix PosixPath to resolve a relative path on root Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 https://github.com/python/cpython/commit/94ad6c674f7687ef22853cb8d42b440d6b42= ddc8 commit: 94ad6c674f7687ef22853cb8d42b440d6b42ddc8 branch: master author: Dong-hee Na committer: =C5=81ukasz Langa date: 2020-08-27T02:24:38+02:00 summary: bpo-33660: Fix PosixPath to resolve a relative path on root files: A Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst M Lib/pathlib.py M Lib/test/test_pathlib.py diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 9f5e27b91178e..babc443dd3b30 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -329,7 +329,10 @@ def _resolve(path, rest): # parent dir path, _, _ =3D path.rpartition(sep) continue - newpath =3D path + sep + name + if path.endswith(sep): + newpath =3D path + name + else: + newpath =3D path + sep + name if newpath in seen: # Already seen this path path =3D seen[newpath] diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 04f7c3d86671b..2cb6738a295b6 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -2349,6 +2349,15 @@ def test_open_mode(self): st =3D os.stat(join('other_new_file')) self.assertEqual(stat.S_IMODE(st.st_mode), 0o644) =20 + def test_resolve_root(self): + current_directory =3D os.getcwd() + try: + os.chdir('/') + p =3D self.cls('spam') + self.assertEqual(str(p.resolve()), '/spam') + finally: + os.chdir(current_directory) + def test_touch_mode(self): old_mask =3D os.umask(0) self.addCleanup(os.umask, old_mask) diff --git a/Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rs= t b/Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst new file mode 100644 index 0000000000000..cce3dbb1c6ea5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-06-12-23-30-41.bpo-33660.AdDn5Z.rst @@ -0,0 +1,2 @@ +Fix pathlib.PosixPath to resolve a relative path located on the root +directory properly.