Mailman 3 June 2022 - Python-Dev

Deadlock when interrupting interpreter initialisation with ptrace?
by Gabriele 07 Jun '22

07 Jun '22

Hi there I hope you don't mind me sharing my experience with testing the austinp variant of Austin with Python >=2.7,<3.11. The austinp variant is a variant of Austin (https://github.com/P403n1x87/austin) for Linux that uses ptrace to seize and interrupt/continue threads to capture native stack traces using libunwind. During testing, I have discovered that there are good chances of causing what looks like a deadlock in Python if the seizing and interrupting of threads happen very early when spawning a Python subprocess from austinp. This seems to coincide with the initialisation of the interpreter when modules are being loaded. To avoid interfering so destructively with Python, I have added a sleep of about 0.5s on fork to prevent sampling during this initialisation phase, which has helped significantly. However, I think this poses one question: is this behaviour from Python to be expected or is it perhaps an indication of a potential bug? Whilst I find it conceivable that something like this could happen, given the locking that happens around imports, is it acceptable that the pausing and resuming of the execution of a thread lead to a potential deadlock? Cheers, Gabriele

4 5

Almost accepting PEP 681 – Data Class Transforms
by Petr Viktorin 07 Jun '22

07 Jun '22

Hello, As an initial implementation that will be improved in the future, the specification in PEP 681 is fine. Feel free to add the decorator to Python 3.11 at your convenience. However, the PEP includes several worrying recommendations like: - we recommend that the maintainers of attrs move away from the legacy semantics and adopt auto_attribs behaviors by default. - We chose not to support this feature and recommend that attrs users avoid converters. - Attrs users should use the dataclass-standard eq and order parameter names instead. These are probably meant as recommendations from typing-sig, but an accepted PEP represents consensus of the entire Python community. A typing PEP is not an appropriate place to make recommendations like this, especially without reaching out to the maintainer of attrs. As far as I know,the attrs and pydantic libraries are using the reference implementation, but their authors weren't consulted on the PEP itself. Could you either change the wording (e.g. say that the unsupported features need bespoke type-checker functionality for proper type checking), or work with attrs to make the same recommendations in its documentation? Happy typing, — Petr, on behalf of the Steering Council

6 7

Looking for volunteers to maintain the multiprocessing and concurrent.futures modules
by Victor Stinner 06 Jun '22

06 Jun '22

Hi, On buildbots, it's common that we get at least one multiprocessing test failure per week. While I just reported a new one, I wanted to add a "expert-multiprocessing" label, but it didn't exist. I just created the label :-) https://github.com/python/cpython/labels/expert-multiprocessing I added the label manually to 240 issues! Oh, they are many open issues about multiprocessing and concurrent.futures modules. Documentation issues, feature requests, but also a long list of real bugs ("hang" is a common term in these reports). Handling multiple processes is hard to get it right, especially when supporting multiple operating systems matter. In Python 3.8, multiprocessing got shared memory support which opened another can of worms :-) Sadly, current multiprocessing maintainers are no longer available and the number of open issues and open pull requests is increasing. I'm not an expert in multiprocessing, but I'm volunteer to mentor someone who is expert and is interested to go through these lists. I cannot dedicate a lot of time, but I will try to be my best to be available :-) See also the GitHub Projects: Multiprocessing issues: https://github.com/orgs/python/projects/14 Pickle and copy issues: https://github.com/orgs/python/projects/9 If you are interested, contact me in private. Sorry, I am not available to teach you how multiprocessing works. I can only assist people who already know it. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 0

Summary of Python tracker Issues
by Python tracker 04 Jun '22

04 Jun '22

ACTIVITY SUMMARY (2022-05-27 - 2022-06-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7146 ( +0) closed 51841 ( +0) total 58987 ( +0) Open issues with patches: 2890 Most recent 15 issues with no replies (15) ========================================== #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin https://bugs.python.org/issue47258 #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47253: LOAD_GLOBAL instruction with wrong source position https://bugs.python.org/issue47253 #47252: socket.makefile documentation is missing data regarding the 'b https://bugs.python.org/issue47252 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47244: email.utils.formataddr does not respect double spaces https://bugs.python.org/issue47244 #47242: Annoying white bar in IDLE (line 457 in sidebar.py) https://bugs.python.org/issue47242 #47241: [C API] Move the PyCodeObject structure to the internal C API https://bugs.python.org/issue47241 #47238: Python threading.Event().wait() depends on the system time https://bugs.python.org/issue47238 #47236: Document types.CodeType.replace() changes about co_exceptionta https://bugs.python.org/issue47236 #47228: Document that naïve datetime objects represent local time https://bugs.python.org/issue47228 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47219: asyncio with two interpreter instances https://bugs.python.org/issue47219 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 Most recent 15 issues waiting for review (15) ============================================= #47256: re: limit the maximum capturing group to 1,073,741,823, reduce https://bugs.python.org/issue47256 #47255: Many broken :meth: roles in the docs https://bugs.python.org/issue47255 #47254: enhanced dir? https://bugs.python.org/issue47254 #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE https://bugs.python.org/issue47251 #47243: Duplicate entry in 'Objects/unicodetype_db.h' https://bugs.python.org/issue47243 #47233: show_caches option affects code positions reported by dis.get_ https://bugs.python.org/issue47233 #47222: subprocess.Popen() should allow capturing output and sending i https://bugs.python.org/issue47222 #47218: adding name to lzmafile https://bugs.python.org/issue47218 #47217: adding name to BZ2File https://bugs.python.org/issue47217 #47216: adding mtime option to gzip open() https://bugs.python.org/issue47216 #47215: Add "unstable" frame stack api https://bugs.python.org/issue47215 #47208: Support libffi implementations that cannot support invocations https://bugs.python.org/issue47208 #47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo https://bugs.python.org/issue47205 #47200: Add ZipInfo.mode property https://bugs.python.org/issue47200 #47199: multiprocessing: micro-optimize Connection.send_bytes() method https://bugs.python.org/issue47199

1 0

[RELEASE] The second Python 3.11 beta (3.11.0b2) is available
by Pablo Galindo Salgado 03 Jun '22

03 Jun '22

Does anyone want bug fixes? Because we have 164 new commits fixing different things, from code to documentation. If you have reported some issue after 3.11.0b1, you should check if is fixed and if not, make sure you tell us so we can take a look. We still have two more betas to go so help us to make sure we don't miss anything so everything is ready for the final release!! https://www.python.org/downloads/release/python-3110b2/ ## This is a beta preview of Python 3.11 Python 3.11 is still in development. 3.11.0b2 is the second of four planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We **strongly encourage** maintainers of third-party Python projects to **test with 3.11** during the beta phase and report issues found to [the Python bug tracker](https://github.com/python/cpython/issues) as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Monday, 2021-08-02). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.11.0rc1, the first release candidate. To achieve that, it will be **extremely important** to get as much exposure for 3.11 as possible during the beta phase. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. # Major new features of the 3.11 series, compared to 3.10 Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * [PEP 680](https://www.python.org/dev/peps/pep-0680/)-- tomllib: Support for Parsing TOML in the Standard Library * [PEP 675](https://www.python.org/dev/peps/pep-0675/)-- Arbitrary Literal String Type * [PEP 655](https://www.python.org/dev/peps/pep-0655/)-- Marking individual TypedDict items as required or potentially-missing * [PEP 681](https://www.python.org/dev/peps/pep-0681/)-- Data Class Transforms * [bpo-46752](https://bugs.python.org/issue46752)-- Introduce task groups to asyncio * [bpo-433030](https://github.com/python/cpython/issues/34627/) -- Atomic grouping ((?>...)) and possessive quantifiers (`*+, ++, ?+, {m,n}+`) are now supported in regular expressions. * The [Faster Cpython Project](https://github.com/faster-cpython/) is already yielding some exciting results. Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.22x speedup on the standard benchmark suite. See [Faster CPython]( https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython) for details. * <small>(Hey, **fellow core developer,** if a feature you find important is missing from this list, [let Pablo know](mailto:pablogsal@python.org ).)</small> The next pre-release of Python 3.11 will be 3.11.0b3, currently scheduled for Thursday, 2022-06-16. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different The Planck time is the time required for light to travel a distance of 1 Planck length in a vacuum, which is a time interval of approximately `5.39*10^(−44)` s. No current physical theory can describe timescales shorter than the Planck time, such as the earliest events after the Big Bang, and it is conjectured that the structure of time breaks down on intervals comparable to the Planck time. While there is currently no known way to measure time intervals on the scale of the Planck time, researchers in 2020 found that the accuracy of an atomic clock is constrained by quantum effects on the order of the Planck time, and for the most precise atomic clocks thus far they calculated that such effects have been ruled out to around `10^−33` s, or 10 orders of magnitude above the Planck scale. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ Regards from sunny London, Pablo Galindo Salgado

4 11

[RELEASE] Expedited release of Python3.11.0b3!!
by Pablo Galindo Salgado 02 Jun '22

02 Jun '22

Hi everyone, Due to a known incompatibility with pytest and the previous beta release (Python 3.11.0b2) and after some deliberation, me and the rest of the release team have decided to do an expedited release of Python 3.11.0b3 so the community can continue testing their packages with pytest and therefore testing the betas as expected. # Where can I get the new release? https://www.python.org/downloads/release/python-3110b3/ # What happened? Pytest by default rewrites the AST nodes in the testing code to provide better diagnostics when something fails in the test. For doing this, it creates new AST nodes that are then compiled. In Python 3.11, after some changes in the compiler and AST nodes, these new AST nodes that pytest was creating were invalid. This causes CPython to crash in debug mode because we have several assert statements in the compiler, but in release mode this doesn't cause always a crash, but it creates potential corrupted structures in the compiler silently. In 3.11.0b3 we changed the compiler to reject invalid AST nodes, so what was a silent problem and a crash in debug mode turned into an exception being raised. We had a fix to allow the nodes that pytest is creating to work to preserve backwards compatibility but unfortunately, it didn't make it into 3.11.0b2. Is still possible to use pytest with 3.11.0b2 if you add "--assert=plain" to the pytest invocation but given how many users would have to modify their test suite invocation we decided to proceed with a new release that has the fix. # What happens with future beta releases Python 3.11.0b3 should be considered as an extra beta release. Instead of four beta releases, we will have five and the next beta release (3.11.0b4) will happen as scheduled on Thursday, 2022-06-16. # We hope you enjoy the new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. https://www.python.org/psf/ If you have any questions, please reach out to me or another member of the release team :) Your friendly release team, Ned Deily @nad https://discuss.python.org/u/nad Steve Dower @steve.dower https://discuss.python.org/u/steve.dower Pablo Galindo Salgado @pablogsal https://discuss.python.org/u/pablogsal

3 2

PEP 689 – Semi-stable C API tier
by Petr Viktorin 01 Jun '22

01 Jun '22

Hello, Victor and others are organizing the C-API to make it clearer what's public and what's private (see Victor's file-based stats: https://pythoncapi.readthedocs.io/stats.html) It became clear that we need a tier between public and private C-API. PEP 689 proposes such API, with an opt-in macro and transition period. Please discuss. And if you can think of a better name, that would be great :) The PEP is at: https://peps.python.org/pep-0689/ Thread where this started: https://mail.python.org/archives/list/python-dev@python.org/thread/MA4FQ7G6… Implementation notes: https://github.com/python/cpython/issues/91744 Here's the current version for easy quoting: ---------- PEP: 689 Title: Semi-stable C API tier Author: Petr Viktorin <encukou(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 523 Created: 22-Apr-2022 Python-Version: 3.11 Abstract ======== Some functions and types of the C-API are designated *semi-stable*, meaning that they will not change in patch (bugfix/security) releases, but may change between minor releases (e.g. between 3.11 and 3.12) without deprecation warnings. Motivation & Rationale ====================== The Python C-API is currently divided into `three tiers <https://devguide.python.org/c-api/>`__: - Limited API, with high compatibility expectations - Public API, which follows the :pep:`backwards compatibility policy <387>`, and requires deprecation warnings before changes - Private (internal) API, which can change at any time. We need a tier between Public and Private to accommodate tools like advanced debuggers and JIT compilers, which need access to CPython internals but assume the C-API they use does not change in patch releases. Setting Stability Expectations ------------------------------ Currently, there are no guarantees for the internal API -- that is, anything that requires ``Py_BUILD_CORE`` or is named with a leading underscore. This API can change without warning at any time, and code that uses it is pinned to a specific build of Python. However, in practice, even the internal API usually happens to be stable in patch releases: - Some CPython core developers take this as an an unwritten rule. - Patch releases only contain bugfixes, which are unlikely to change the API. Semi-stable API will make the stability expectations more explicit. It will also hopefully encourage existing users of the private API to reach out to python-dev, so we can expose, standardize and test an API for some of their use cases. Reserving underscores for Private API ------------------------------------- :pep:`523` introduced functions for use by debuggers and JIT compilers, which are stable only across minor releases. The functions names have leading underscores to suggest their limited stability. However, leading underscores usually mark *fully private* API. CPython developers familiar with the “underscore means internal” convention are unlikely to check if underscored functions they are changing are documented and used outside CPython itself. This proposal brings us a bit closer to reserving underscores only for truly private, unstable, hands-off API. Warning about API that is changed often --------------------------------------- The ``PyCode_New()`` family is an example of functions that are documented as unstable, and also often change in practice. Moving it to the semi-stable tier will make its status obvious even to people who don't read the docs carefully enough, and will make it hard to use accidentally. Changes during the Beta period ------------------------------ Since the API itself can change continuously up until Beta 1 (feature freeze) of a minor version, major users of this API are unlikely to test Alpha releases and provide feedback. It is very difficult to determine what needs to be exposed as semi-stable. Additions to the semi-stable tier will count as *stabilization*, and will be allowed up to Release Candidate 1. Specification ============= Several functions and types (“APIs”) will be moved to a new *semi-stable* tier. They will be expected to stay stable across patch releases, but may change or be removed without warning in minor releases (3.x.0), including Alpha and Beta releases of 3.x.0. When they change significantly, code that uses them should no longer compile (e.g. arguments should be added/removed, or a function should be renamed, but the semantic meaning of an argument should not change). Their definitions will be moved to a new directory, ``Include/semistable/``, and will be included from ``Python.h``. From Python 3.12 on, these APIs will only be usable when the ``Py_USING_SEMI_STABLE_API`` macro is defined. CPython will only define the macro for building CPython itself (``Py_BUILD_CORE``). To make transition to semi-stable API easier, in Python 3.11 the APIs will be available without ``Py_USING_SEMI_STABLE_API`` defined. In this case, using them will generate a deprecation warning on compilers that support ``Py_DEPRECATED``. A similar deprecation period will be used when making more APIs semi-stable in the future: - When moving from public API, the deprecation period should follow Python's backwards compatibility policy (currently, it should last at least two releases). - When moving from public API that is documented as unstable, the deprecation period can only last one release. - When moving from private API or adding new API, no deprecation period is necessary. Leading underscores will be removed from the names of the moved APIs. The old underscored name of a renamed API will be available (as an alias using ``#define``) at least until that API changes. The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be documented, and documentation of each semi-stable API will be updated. Adjustments during Beta periods ------------------------------- New APIs can be added to the semi-stable tier, and private APIs can be moved to it, up to the first release candidate of a new minor version. Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta period. In the Beta period, no API may be moved to more private tier, e.g. what is public in Beta 1 must stay public until the final release. Initial semi-stable API ----------------------- The following API will initially be semi-stable. The set may be adjusted for 3.11. Code object constructors: - ``PyCode_New()`` - ``PyCode_NewWithPosOnlyArgs()`` Frame evaluation API (PEP 523): - ``_PyFrameEvalFunction`` - ``_PyInterpreterState_GetEvalFrameFunc()`` - ``_PyInterpreterState_SetEvalFrameFunc()`` - ``_PyEval_RequestCodeExtraIndex()`` - ``_PyCode_GetExtra()`` - ``_PyCode_SetExtra()`` - ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct) - ``_PyFrame_GetFrameObject`` - ``PyEval_EvalFrameDefault`` (new function that calls ``_PyEval_EvalFrameDefault``, but takes ``PyFrameObject`` rather than ``_PyInterpreterFrame``) (Leading underscores will be removed as mentioned above.) Backwards Compatibility ======================= The C API backwards compatibility story will be made clearer. How to Teach This ================= The changes affect advanced C programmers, who should consult the updated reference documentation, devguide and/or What's New document·. Reference Implementation ======================== https://github.com/python/cpython/issues/91744 Rejected Ideas ============== It might be good to add a similar tier in the Python (not C) API, e.g. for ``types.CodeType``. However, the opt-in mechanism would need to be different (if any). This is outside the scope of the PEP. Open Issues =========== - “Semi-stable” is not a perfect name. - The exact set of exposed API may change. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

13 24