Mailman 3 March 2022 - Python-Dev

[RELEASE] Python 3.10.3, 3.9.11, 3.8.13, and 3.7.13 are now available with security content
by Łukasz Langa 16 Mar '22

16 Mar '22

Welcome again to the exciting world of releasing new Python versions! Last time around I was complaining about cursed releases <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>. This time around I could complain about security content galore and how one of them <https://mta.openssl.org/pipermail/openssl-announce/2022-March/000216.html> ruined my ingenious idea to release Python on Pi Day and call it Py Day <https://discuss.python.org/t/py-day-is-coming-a-joint-security-release-spre…>. Well, you can’t have everything in life. Or at least not everything at once. And yet it seems this time around we’ve got a lot of security fixes all at once. Just look at this list: 15 (sic!) CVEs: libexpat upgraded from 2.4.1 to 2.4.7 (BPO-46794 <https://bugs.python.org/issue46794>, BPO-46932 <https://bugs.python.org/issue46932>, BPO-46811 <https://bugs.python.org/issue46811>, BPO-46784 <https://bugs.python.org/issue46784>, BPO-46400 <https://bugs.python.org/issue46400>) CVE-2022-0778: OpenSSL upgraded from 1.1.1l to 1.1.1n in macOS and Windows installers (BPO-47024 <https://bugs.python.org/issue47024>) CVE-2016-3189, CVE-2019-12900: bzip2 upgraded from 1.0.6 to 1.0.8 in Windows installers (BPO-44549 <https://bugs.python.org/issue44549>) CVE-2022-26488: Windows installer now ensures the correct path is being repaired when “Add to PATH” is used (BPO-46948 <https://bugs.python.org/issue46948>) CVE-2021-28363: bundled pip upgraded from 21.2.4 to 22.0.4 (BPO-46985 <https://bugs.python.org/issue46985>) authorization bypass fixed in urllib.request (BPO-46756 <https://bugs.python.org/issue46756>) REDoS avoided in importlib.metadata (BPO-46474 <https://bugs.python.org/issue46474>) SQLite upgraded from 3.36.0 to 3.37.2 in macOS and Windows installers (BPO-45925 <https://bugs.python.org/issue45925>) <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.10.3 Get it here: https://www.python.org/downloads/release/python-3103/ <https://www.python.org/downloads/release/python-3103/> Python 3.10.3 is the third maintenance release of the newest version of the Python programming language, which contains many new features and optimizations. We recommend it over the other releases listed below. This is a large bugfix release with 220 commits since 3.10.2. Just look at the change log <https://docs.python.org/release/3.10.3/whatsnew/changelog.html>! The next maintenance release of Python 3.10 is planned for early June. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.9.11 Get it here: https://www.python.org/downloads/release/python-3911/ <https://www.python.org/downloads/release/python-3911/> This is the penultimate planned full bugfix release of Python 3.9. In mid-May, we’ll be releasing the last one, after which the 3.9 series will enter its security-only fixes period. More details in PEP 596 <https://www.python.org/dev/peps/pep-0596/>. Python 3.9 is the first Python version since 2.7 to have a regular bugfix release larger than “.10”. It’s also still a significant release at 163 commits since 3.9.10. That’s in fact 30+ commits more than between 3.9.9 and 3.9.10. The change log <https://docs.python.org/release/3.9.11/whatsnew/changelog.html> isn’t as long as the 3.10.3 one but it’s still pretty extensive! As a reminder, on macOS, the default installer is now the new universal2 variant. It’s compatible with Mac OS X 10.9 and newer, including macOS 11 Big Sur and macOS 12 Monterey. Python installed with this variant will work natively on Apple Silicon processors. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.8.13 Get it here: https://www.python.org/downloads/release/python-3813/ <https://www.python.org/downloads/release/python-3813/> Changes <https://docs.python.org/release/3.8.13/whatsnew/changelog.html> here are almost exclusively security-only as the life cycle of Python versions prescribes. You might have noticed there is a small number of regular bug fixes nonetheless. This is because without those we wouldn’t be able to continue running the full test suite for the 3.8 branch. This in turn could hide regressions in future security fixes. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>Python 3.7.13 Get it here: https://www.python.org/downloads/release/python-3713/ <https://www.python.org/downloads/release/python-3713/> Just like 3.8, Python 3.7 is in its security-only fixes period. In turn, the changes in 3.7.13 <https://docs.python.org/release/3.7.13/whatsnew/changelog.html> look almost identical to the ones in 3.8.13. Python 3.7 will continue to receive source-only releases until June 2023. <https://discuss.python.org/t/python-3-10-3-3-9-11-3-8-13-and-3-7-13-are-now…>We hope you enjoy the new releases Your friendly release team, Łukasz Langa @ambv <https://discuss.python.org/u/ambv> Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower>

3 3

Python 3.9.11
by Prasad, PCRaghavendra 16 Mar '22

16 Mar '22

Hi Team, Can someone please let us know the release date of Python 3.9.11 ( with libexpat 2.4.8 security issues fixed ) In the python.org releases it was mentioned as 14-march-2022, but still, I couldn't see the bin/source code. Can someone help with this Thanks, Raghavendra Internal Use - Confidential

5 4

Gathering edge cases around type annotation PEPs
by Brett Cannon 16 Mar '22

16 Mar '22

I created https://discuss.python.org/t/finding-edge-cases-for-peps-484-463-and-649-ty… on behalf of the SC to help gather edge cases where the various approaches that PEPs have proposed will fail. Our hope is to get an overall picture of the trade-offs the various PEPs ask us to make. PLEASE DO NOT REPLY HERE! I'm reaching out to the typing SIG and Pydantic (at least) and it's much easier for others to log into discuss.python.org with a GitHub account than it is to sign up for this mailing list just to participate in this discussion.

1 0

Defining tiered platform support
by Brett Cannon 16 Mar '22

16 Mar '22

Do we officially support NetBSD? Do you know how to find out if we do? You might think to look at https://www.python.org/dev/peps/pep-0011/#supporting-platforms , but that just loosely defines the criteria and it still doesn't list the actual platforms we support. (BTW I don't know if we do officially support NetBSD, hence this email.) I think we should clarify this sort of thing and be a bit more upfront with the level of support we expect/provide for a platform. As such, I want to restructure PEP 11 to list the platforms we support, not just list the platforms we stopped supporting. To do this I want define 3 different tiers that outline what our support requirements and promises are for platforms. Tier 1 is the stuff we run CI against: latest Windows, latest macOS, Linux w/ the latest glibc (I don't know of a better way to define Linux support as I don't know if a per-distro list is the right abstraction). These are platforms we won't even let code be committed for if they would break; they block releases if they don't work. These platforms we all implicitly promise to support. Tier 2 is the platforms we would revert a change within 24 hours if they broke: latest FeeBSD, older Windows, older macOS, Linux w/ older glibc.This is historically the "stable buildbot plus a core dev" group of platforms. The change I would like to see is two core devs (in case one is on vacation), and a policy as to how a platform ends up here (e.g. SC must okay it based on consensus of everyone). The stable buildbot would still be needed to know if a release is blocked as we would hold a release up if they were red. The platform and the core devs supporting these platforms would be listed in PEP 11. Tier 3 are platforms we accept PRs for to keep working, but they won't block a release. At least one core dev must be listed to be in charge of PRs that affect the platform. A buildbot would be nice but not required. I'm thinking either historical platforms we support or something like Emscripten that's working towards being a tier 2 platform. I'm not sure if we want to have explicit approval to be in this tier beyond the outlined requirements, but obviously if a core dev isn't keeping up with PRs then the platform will be dropped. All other platforms we do not directly support in the repository and it is up to the community to keep functioning. We can obviously keep accepting general patches to up compatibility, but platform-specific patches for anything outside of these tiers wouldn't. No issue if someone provides a buildbot for their own benefit, but these buildbots would never hold anything up. When a Python version hits b1, then that platform is considered supported for that version as long as the requirements outlined above continue to be met. Once the Python version hits EOL then like anything, support ends no matter what. We could capture the supported platforms for a version in the release schedule PEP if people want. I would expect appropriate labels being added to the builders listed at https://buildbot.python.org/all/#/builders to signify which platforms we block releases for (e.g. `tier-1,`, `tier-2`, or `release-blocker` as a generic label). I would expect PEP 11 to list the appropriate C symbol that's set for that platform, e.g. __linux__. I don't know if we want to bother listing CPU architectures since we are a pure C project which makes CPU architecture less of a thing, but I'm personally open to the idea of CPU architectures being a part of the platform definition. I don't think we should cover C compilers here as that's covered by PEP 7. Otherwise PEP 7 should only list C versions/features and PEP 11 lists compilers and their versions. FYI the tier support idea and the rough definitions above come from Rust: https://doc.rust-lang.org/nightly/rustc/platform-support.html .

9 19

PEP 684: A Per-Interpreter GIL
by Eric Snow 15 Mar '22

15 Mar '22

I'd really appreciate feedback on this new PEP about making the GIL per-interpreter. The PEP targets 3.11, but we'll see if that is too close. I don't mind waiting one more release, though I'd prefer 3.11 (obviously). Regardless, I have no intention of rushing this through at the expense of cutting corners. Hence, we'll see how it goes. The PEP text is included inline below. Thanks! -eric =================================================== PEP: 684 Title: A Per-Interpreter GIL Author: Eric Snow <ericsnowcurrently(a)gmail.com> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 08-Mar-2022 Python-Version: 3.11 Post-History: 08-Mar-2022 Resolution: Abstract ======== Since Python 1.5 (1997), CPython users can run multiple interpreters in the same process. However, interpreters in the same process have always shared a significant amount of global state. This is a source of bugs, with a growing impact as more and more people use the feature. Furthermore, sufficient isolation would facilitate true multi-core parallelism, where interpreters no longer share the GIL. The changes outlined in this proposal will result in that level of interpreter isolation. High-Level Summary ================== At a high level, this proposal changes CPython in the following ways: * stops sharing the GIL between interpreters, given sufficient isolation * adds several new interpreter config options for isolation settings * adds some public C-API for fine-grained control when creating interpreters * keeps incompatible extensions from causing problems The GIL ------- The GIL protects concurrent access to most of CPython's runtime state. So all that GIL-protected global state must move to each interpreter before the GIL can. (In a handful of cases, other mechanisms can be used to ensure thread-safe sharing instead, such as locks or "immortal" objects.) CPython Runtime State --------------------- Properly isolating interpreters requires that most of CPython's runtime state be stored in the ``PyInterpreterState`` struct. Currently, only a portion of it is; the rest is found either in global variables or in ``_PyRuntimeState``. Most of that will have to be moved. This directly coincides with an ongoing effort (of many years) to greatly reduce internal use of C global variables and consolidate the runtime state into ``_PyRuntimeState`` and ``PyInterpreterState``. (See `Consolidating Runtime Global State`_ below.) That project has `significant merit on its own <Benefits to Consolidation_>`_ and has faced little controversy. So, while a per-interpreter GIL relies on the completion of that effort, that project should not be considered a part of this proposal--only a dependency. Other Isolation Considerations ------------------------------ CPython's interpreters must be strictly isolated from each other, with few exceptions. To a large extent they already are. Each interpreter has its own copy of all modules, classes, functions, and variables. The CPython C-API docs `explain further <caveats_>`_. .. _caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats However, aside from what has already been mentioned (e.g. the GIL), there are a couple of ways in which interpreters still share some state. First of all, some process-global resources (e.g. memory, file descriptors, environment variables) are shared. There are no plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take multiple interpreters into account. This includes CPython's runtime and the stdlib, as well as extension modules that rely on global variables. Bugs should be opened in these cases, as some already have been. Depending on Immortal Objects ----------------------------- :pep:`683` introduces immortal objects as a CPython-internal feature. With immortal objects, we can share any otherwise immutable global objects between all interpreters. Consequently, this PEP does not need to address how to deal with the various objects `exposed in the public C-API <capi objects_>`_. It also simplifies the question of what to do about the builtin static types. (See `Global Objects`_ below.) Both issues have alternate solutions, but everything is simpler with immortal objects. If PEP 683 is not accepted then this one will be updated with the alternatives. This lets us reduce noise in this proposal. Motivation ========== The fundamental problem we're solving here is a lack of true multi-core parallelism (for Python code) in the CPython runtime. The GIL is the cause. While it usually isn't a problem in practice, at the very least it makes Python's multi-core story murky, which makes the GIL a consistent distraction. Isolated interpreters are also an effective mechanism to support certain concurrency models. :pep:`554` discusses this in more detail. Indirect Benefits ----------------- Most of the effort needed for a per-interpreter GIL has benefits that make those tasks worth doing anyway: * makes multiple-interpreter behavior more reliable * has led to fixes for long-standing runtime bugs that otherwise hadn't been prioritized * has been exposing (and inspiring fixes for) previously unknown runtime bugs * has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) * has driven cleaner and more complete runtime finalization * led to structural layering of the C-API (e.g. ``Include/internal``) * also see `Benefits to Consolidation`_ below Furthermore, much of that work benefits other CPython-related projects: * performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram) * extension module isolation (see :pep:`630`, etc.) * embedding CPython Existing Use of Multiple Interpreters ------------------------------------- The C-API for multiple interpreters has been used for many years. However, until relatively recently the feature wasn't widely known, nor extensively used (with the exception of mod_wsgi). In the last few years use of multiple interpreters has been increasing. Here are some of the public projects using the feature currently: * `mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_ * `OpenStack Ceph <https://github.com/ceph/ceph/pull/14971>`_ * `JEP <https://github.com/ninia/jep>`_ * `Kodi <https://github.com/xbmc/xbmc>`_ Note that, with :pep:`554`, multiple interpreter usage would likely grow significantly (via Python code rather than the C-API). PEP 554 ------- :pep:`554` is strictly about providing a minimal stdlib module to give users access to multiple interpreters from Python code. In fact, it specifically avoids proposing any changes related to the GIL. Consider, however, that users of that module would benefit from a per-interpreter GIL, which makes PEP 554 more appealing. Rationale ========= During initial investigations in 2014, a variety of possible solutions for multi-core Python were explored, but each had its drawbacks without simple solutions: * the existing practice of releasing the GIL in extension modules * doesn't help with Python code * other Python implementations (e.g. Jython, IronPython) * CPython dominates the community * remove the GIL (e.g. gilectomy, "no-gil") * too much technical risk (at the time) * Trent Nelson's "PyParallel" project * incomplete; Windows-only at the time * ``multiprocessing`` * too much work to make it effective enough; high penalties in some situations (at large scale, Windows) * other parallelism tools (e.g. dask, ray, MPI) * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) * this can only end in tears Even in 2014, it was fairly clear that a solution using isolated interpreters did not have a high level of technical risk and that most of the work was worth doing anyway. (The downside was the volume of work to be done.) Specification ============= As `summarized above <High-Level Summary_>`__, this proposal involves the following changes, in the order they must happen: 1. `consolidate global runtime state <Consolidating Runtime Global State_>`_ (including objects) into ``_PyRuntimeState`` 2. move nearly all of the state down into ``PyInterpreterState`` 3. finally, move the GIL down into ``PyInterpreterState`` 4. everything else * add to the public C-API * implement restrictions in ``ExtensionFileLoader`` * work with popular extension maintainers to help with multi-interpreter support Per-Interpreter State --------------------- The following runtime state will be moved to ``PyInterpreterState``: * all global objects that are not safely shareable (fully immutable) * the GIL * mutable, currently protected by the GIL * mutable, currently protected by some other per-interpreter lock * mutable, may be used independently in different interpreters * all other mutable (or effectively mutable) state not otherwise excluded below Furthermore, a number of parts of the global state have already been moved to the interpreter, such as GC, warnings, and atexit hooks. The following state will not be moved: * global objects that are safely shareable, if any * immutable, often ``const`` * treated as immutable * related to CPython's ``main()`` execution * related to the REPL * set during runtime init, then treated as immutable * mutable, protected by some global lock * mutable, atomic Note that currently the allocators (see ``Objects/obmalloc.c``) are shared between all interpreters, protected by the GIL. They will need to move to each interpreter (or a global lock will be needed). This is the highest risk part of the work to isolate interpreters and may require more than just moving fields down from ``_PyRuntimeState``. Some of the complexity is reduced if CPython switches to a thread-safe allocator like mimalloc. .. _proposed capi: C-API ----- The following private API will be made public: * ``_PyInterpreterConfig`` * ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) The following fields will be added to ``PyInterpreterConfig``: * ``own_gil`` - (bool) create a new interpreter lock (instead of sharing with the main interpreter) * ``strict_extensions`` - fail import in this interpreter for incompatible extensions (see `Restricting Extension Modules`_) Restricting Extension Modules ----------------------------- Extension modules have many of the same problems as the runtime when state is stored in global variables. :pep:`630` covers all the details of what extensions must do to support isolation, and thus safely run in multiple interpreters at once. This includes dealing with their globals. Extension modules that do not implement isolation will only run in the main interpreter. In all other interpreters, the import will raise ``ImportError``. This will be done through ``importlib._bootstrap_external.ExtensionFileLoader``. We will work with popular extensions to help them support use in multiple interpreters. This may involve adding to CPython's public C-API, which we will address on a case-by-case basis. Extension Module Compatibility '''''''''''''''''''''''''''''' As noted in `Extension Modules`_, many extensions work fine in multiple interpreters without needing any changes. The import system will still fail if such a module doesn't explicitly indicate support. At first, not many extension modules will, so this is a potential source of frustration. We will address this by adding a context manager to temporarily disable the check on multiple interpreter support: ``importlib.util.allow_all_extensions()``. Documentation ------------- The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be updated with the added API. Impact ====== Backwards Compatibility ----------------------- No behavior or APIs are intended to change due to this proposal, with one exception noted in `the next section <Extension Modules_>`_. The existing C-API for managing interpreters will preserve its current behavior, with new behavior exposed through new API. No other API or runtime behavior is meant to change, including compatibility with the stable ABI. See `Objects Exposed in the C-API`_ below for related discussion. Extension Modules ''''''''''''''''' Currently the most common usage of Python, by far, is with the main interpreter running by itself. This proposal has zero impact on extension modules in that scenario. Likewise, for better or worse, there is no change in behavior under multiple interpreters created using the existing ``Py_NewInterpreter()``. Keep in mind that some extensions already break when used in multiple interpreters, due to keeping module state in global variables. They may crash or, worse, experience inconsistent behavior. That was part of the motivation for :pep:`630` and friends, so this is not a new situation nor a consequence of this proposal. In contrast, when the `proposed API <proposed capi_>`_ is used to create multiple interpreters, the default behavior will change for some extensions. In that case, importing an extension will fail (outside the main interpreter) if it doesn't indicate support for multiple interpreters. For extensions that already break in multiple interpreters, this will be an improvement. Now we get to the break in compatibility mentioned above. Some extensions are safe under multiple interpreters, even though they haven't indicated that. Unfortunately, there is no reliable way for the import system to infer that such an extension is safe, so importing them will still fail. This case is addressed in `Extension Module Compatibility`_ below. Extension Module Maintainers ---------------------------- One related consideration is that a per-interpreter GIL will likely drive increased use of multiple interpreters, particularly if :pep:`554` is accepted. Some maintainers of large extension modules have expressed concern about the increased burden they anticipate due to increased use of multiple interpreters. Specifically, enabling support for multiple interpreters will require substantial work for some extension modules. To add that support, the maintainer(s) of such a module (often volunteers) would have to set aside their normal priorities and interests to focus on compatibility (see :pep:`630`). Of course, extension maintainers are free to not add support for use in multiple interpreters. However, users will increasingly demand such support, especially if the feature grows in popularity. Either way, the situation can be stressful for maintainers of such extensions, particularly when they are doing the work in their spare time. The concerns they have expressed are understandable, and we address the partial solution in `Restricting Extension Modules`_ below. Alternate Python Implementations -------------------------------- Other Python implementation are not required to provide support for multiple interpreters in the same process (though some do already). Security Implications --------------------- There is no known impact to security with this proposal. Maintainability --------------- On the one hand, this proposal has already motivated a number of improvements that make CPython *more* maintainable. That is expected to continue. On the other hand, the underlying work has already exposed various pre-existing defects in the runtime that have had to be fixed. That is also expected to continue as multiple interpreters receive more use. Otherwise, there shouldn't be a significant impact on maintainability, so the net effect should be positive. Performance ----------- The work to consolidate globals has already provided a number of improvements to CPython's performance, both speeding it up and using less memory, and this should continue. Performance benefits to a per-interpreter GIL have not been explored. At the very least, it is not expected to make CPython slower (as long as interpreters are sufficiently isolated). How to Teach This ================= This is an advanced feature for users of the C-API. There is no expectation that this will be taught. That said, if it were taught then it would boil down to the following: In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() to create an interpreter. The config you pass it indicates how you want that interpreter to behave. Reference Implementation ======================== <TBD> Open Issues =========== * What are the risks/hurdles involved with moving the allocators? * Is ``allow_all_extensions`` the best name for the context manager? Deferred Functionality ====================== * ``PyInterpreterConfig`` option to always run the interpreter in a new thread * ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter and only run in that thread Rejected Ideas ============== <TBD> Extra Context ============= Sharing Global Objects ---------------------- We are sharing some global objects between interpreters. This is an implementation detail and relates more to `globals consolidation <Consolidating Runtime Global State>`_ than to this proposal, but it is a significant enough detail to explain here. The alternative is to share no objects between interpreters, ever. To accomplish that, we'd have to sort out the fate of all our static types, as well as deal with compatibility issues for the many objects `exposed in the public C-API <capi objects_>`_. That approach introduces a meaningful amount of extra complexity and higher risk, though prototyping has demonstrated valid solutions. Also, it would likely result in a performance penalty. `Immortal objects <Depending on Immortal Objects_>`_ allow us to share the otherwise immutable global objects. That way we avoid the extra costs. .. _capi objects: Objects Exposed in the C-API '''''''''''''''''''''''''''' The C-API (including the limited API) exposes all the builtin types, including the builtin exceptions, as well as the builtin singletons. The exceptions are exposed as ``PyObject *`` but the rest are exposed as the static values rather than pointers. This was one of the few non-trivial problems we had to solve for per-interpreter GIL. With immortal objects this is a non-issue. Consolidating Runtime Global State ---------------------------------- As noted in `CPython Runtime State`_ above, there is an active effort (separate from this PEP) to consolidate CPython's global state into the ``_PyRuntimeState`` struct. Nearly all the work involves moving that state from global variables. The project is particularly relevant to this proposal, so below is some extra detail. Benefits to Consolidation ''''''''''''''''''''''''' Consolidating the globals has a variety of benefits: * greatly reduces the number of C globals (best practice for C code) * the move draws attention to runtime state that is unstable or broken * encourages more consistency in how runtime state is used * makes multiple-interpreter behavior more reliable * leads to fixes for long-standing runtime bugs that otherwise haven't been prioritized * exposes (and inspires fixes for) previously unknown runtime bugs * facilitates cleaner runtime initialization and finalization * makes it easier to discover/identify CPython's runtime state * makes it easier to statically allocate runtime state in a consistent way * better memory locality for runtime state * structural layering of the C-API (e.g. ``Include/internal``) Furthermore, much of that work benefits other CPython-related projects: * performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram) * extension module isolation (see :pep:`630`, etc.) * embedding CPython Scale of Work ''''''''''''' The number of global variables to be moved is large enough to matter, but most are Python objects that can be dealt with in large groups (like ``Py_IDENTIFIER``). In nearly all cases, moving these globals to the interpreter is highly mechanical. That doesn't require cleverness but instead requires someone to put in the time. State To Be Moved ''''''''''''''''' The remaining global variables can be categorized as follows: * global objects * static types (incl. exception types) * non-static types (incl. heap types, structseq types) * singletons (static) * singletons (initialized once) * cached objects * non-objects * will not (or unlikely to) change after init * only used in the main thread * initialized lazily * pre-allocated buffers * state Those globals are spread between the core runtime, the builtin modules, and the stdlib extension modules. For a breakdown of the remaining globals, run: .. code-block:: bash ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv Already Completed Work '''''''''''''''''''''' As mentioned, this work has been going on for many years. Here are some of the things that have already been done: * cleanup of runtime initialization (see :pep:`432` / :pep:`587`) * extension module isolation machinery (see :pep:`384` / :pep:`3121` / :pep:`489`) * isolation for many builtin modules * isolation for many stdlib extension modules * addition of ``_PyRuntimeState`` * no more ``_Py_IDENTIFIER()`` * statically allocated: * empty string * string literals * identifiers * latin-1 strings * length-1 bytes * empty tuple Tooling ''''''' As already indicated, there are several tools to help identify the globals and reason about them. * ``Tools/c-analyzer/cpython/globals-to-fix.tsv`` - the list of remaining globals * ``Tools/c-analyzer/c-analyzer.py`` * ``analyze`` - identify all the globals * ``check`` - fail if there are any unsupported globals that aren't ignored * ``Tools/c-analyzer/table-file.py`` - summarize the known globals Also, the check for unsupported globals is incorporated into CI so that no new globals are accidentally added. Global Objects '''''''''''''' Global objects that are safe to be shared (without a GIL) between interpreters can stay on ``_PyRuntimeState``. Not only must the object be effectively immutable (e.g. singletons, strings), but not even the refcount can change for it to be safe. Immortality (:pep:`683`) provides that. (The alternative is that no objects are shared, which adds significant complexity to the solution, particularly for the objects `exposed in the public C-API <capi objects_>`_.) Builtin static types are a special case of global objects that will be shared. They are effectively immutable except for one part: ``__subclasses__`` (AKA ``tp_subclasses``). We expect that nothing else on a builtin type will change, even the content of ``__dict__`` (AKA ``tp_dict``). ``__subclasses__`` for the builtin types will be dealt with by making it a getter that does a lookup on the current ``PyInterpreterState`` for that type. References ========== Related: * :pep:`384` * :pep:`432` * :pep:`489` * :pep:`554` * :pep:`573` * :pep:`587` * :pep:`630` * :pep:`683` * :pep:`3121` Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

6 10

PEP 683: "Immortal Objects, Using a Fixed Refcount" (round 3)
by Eric Snow 14 Mar '22

14 Mar '22

I've updated PEP 683 for the feedback I've gotten. Thanks again for that! The updated PEP text is included below. The largest changes involve either the focus of the PEP (internal mechanism to mark objects immortal) or the possible ways that things can break on older 32-bit stable ABI extensions. All other changes are smaller. Given the last round of discussion, I'm hoping this will be the last round before we go to the steering council. -eric ---------------- PEP: 683 Title: Immortal Objects, Using a Fixed Refcount Author: Eric Snow <ericsnowcurrently(a)gmail.com>, Eddie Elizondo <eduardo.elizondorueda(a)gmail.com> Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/TPLEYDCX… Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2022 Python-Version: 3.11 Post-History: 15-Feb-2022, 19-Feb-2022, 28-Feb-2022 Resolution: Abstract ======== Currently the CPython runtime maintains a `small amount of mutable state <Runtime Object State_>`_ in the allocated memory of each object. Because of this, otherwise immutable objects are actually mutable. This can have a large negative impact on CPU and memory performance, especially for approaches to increasing Python's scalability. This proposal mandates that, internally, CPython will support marking an object as one for which that runtime state will no longer change. Consequently, such an object's refcount will never reach 0, and so the object will never be cleaned up. We call these objects "immortal". (Normally, only a relatively small number of internal objects will ever be immortal.) The fundamental improvement here is that now an object can be truly immutable. Scope ----- Object immortality is meant to be an internal-only feature. So this proposal does not include any changes to public API or behavior (with one exception). As usual, we may still add some private (yet publicly accessible) API to do things like immortalize an object or tell if one is immortal. Any effort to expose this feature to users would need to be proposed separately. There is one exception to "no change in behavior": refcounting semantics for immortal objects will differ in some cases from user expectations. This exception, and the solution, are discussed below. Most of this PEP focuses on an internal implementation that satisfies the above mandate. However, those implementation details are not meant to be strictly proscriptive. Instead, at the least they are included to help illustrate the technical considerations required by the mandate. The actual implementation may deviate somewhat as long as it satisfies the constraints outlined below. Furthermore, the acceptability of any specific implementation detail described below does not depend on the status of this PEP, unless explicitly specified. For example, the particular details of: * how to mark something as immortal * how to recognize something as immortal * which subset of functionally immortal objects are marked as immortal * which memory-management activities are skipped or modified for immortal objects are not only CPython-specific but are also private implementation details that are expected to change in subsequent versions. Implementation Summary ---------------------- Here's a high-level look at the implementation: If an object's refcount matches a very specific value (defined below) then that object is treated as immortal. The CPython C-API and runtime will not modify the refcount (or other runtime state) of an immortal object. Aside from the change to refcounting semantics, there is one other possible negative impact to consider. A naive implementation of the approach described below makes CPython roughly 4% slower. However, the implementation is performance-neutral once known mitigations are applied. Motivation ========== As noted above, currently all objects are effectively mutable. That includes "immutable" objects like ``str`` instances. This is because every object's refcount is frequently modified as the object is used during execution. This is especially significant for a number of commonly used global (builtin) objects, e.g. ``None``. Such objects are used a lot, both in Python code and internally. That adds up to a consistent high volume of refcount changes. The effective mutability of all Python objects has a concrete impact on parts of the Python community, e.g. projects that aim for scalability like Instragram or the effort to make the GIL per-interpreter. Below we describe several ways in which refcount modification has a real negative effect on such projects. None of that would happen for objects that are truly immutable. Reducing CPU Cache Invalidation ------------------------------- Every modification of a refcount causes the corresponding CPU cache line to be invalidated. This has a number of effects. For one, the write must be propagated to other cache levels and to main memory. This has small effect on all Python programs. Immortal objects would provide a slight relief in that regard. On top of that, multi-core applications pay a price. If two threads (running simultaneously on distinct cores) are interacting with the same object (e.g. ``None``) then they will end up invalidating each other's caches with each incref and decref. This is true even for otherwise immutable objects like ``True``, ``0``, and ``str`` instances. CPython's GIL helps reduce this effect, since only one thread runs at a time, but it doesn't completely eliminate the penalty. Avoiding Data Races ------------------- Speaking of multi-core, we are considering making the GIL a per-interpreter lock, which would enable true multi-core parallelism. Among other things, the GIL currently protects against races between multiple concurrent threads that may incref or decref the same object. Without a shared GIL, two running interpreters could not safely share any objects, even otherwise immutable ones like ``None``. This means that, to have a per-interpreter GIL, each interpreter must have its own copy of *every* object. That includes the singletons and static types. We have a viable strategy for that but it will require a meaningful amount of extra effort and extra complexity. The alternative is to ensure that all shared objects are truly immutable. There would be no races because there would be no modification. This is something that the immortality proposed here would enable for otherwise immutable objects. With immortal objects, support for a per-interpreter GIL becomes much simpler. Avoiding Copy-on-Write ---------------------- For some applications it makes sense to get the application into a desired initial state and then fork the process for each worker. This can result in a large performance improvement, especially memory usage. Several enterprise Python users (e.g. Instagram, YouTube) have taken advantage of this. However, the above refcount semantics drastically reduce the benefits and have led to some sub-optimal workarounds. Also note that "fork" isn't the only operating system mechanism that uses copy-on-write semantics. Anything that uses ``mmap`` relies on copy-on-write, including sharing data from shared object files between processes. Rationale ========= The proposed solution is obvious enough that both of this proposal's authors came to the same conclusion (and implementation, more or less) independently. The Pyston project `uses a similar approach <Pyston_>`_. Other designs were also considered. Several possibilities have also been discussed on python-dev in past years. Alternatives include: * use a high bit to mark "immortal" but do not change ``Py_INCREF()`` * add an explicit flag to objects * implement via the type (``tp_dealloc()`` is a no-op) * track via the object's type object * track with a separate table Each of the above makes objects immortal, but none of them address the performance penalties from refcount modification described above. In the case of per-interpreter GIL, the only realistic alternative is to move all global objects into ``PyInterpreterState`` and add one or more lookup functions to access them. Then we'd have to add some hacks to the C-API to preserve compatibility for the may objects exposed there. The story is much, much simpler with immortal objects Impact ====== Benefits -------- Most notably, the cases described in the above examples stand to benefit greatly from immortal objects. Projects using pre-fork can drop their workarounds. For the per-interpreter GIL project, immortal objects greatly simplifies the solution for existing static types, as well as objects exposed by the public C-API. In general, a strong immutability guarantee for objects enables Python applications to scale like never before. This is because they can then leverage multi-core parallelism without a tradeoff in memory usage. This is reflected in most of the above cases. Performance ----------- A naive implementation shows `a 4% slowdown`_. We have demonstrated a return to performance-neutral with a handful of basic mitigations applied. See the `mitigation`_ section below. On the positive side, immortal objects save a significant amount of memory when used with a pre-fork model. Also, immortal objects provide opportunities for specialization in the eval loop that would improve performance. .. _a 4% slowdown: https://github.com/python/cpython/pull/19474#issuecomment-1032944709 Backward Compatibility ---------------------- Ideally this internal-only feature would be completely compatible. However, it does involve a change to refcount semantics in some cases. Only immortal objects are affected, but this includes high-use objects like ``None``, ``True``, and ``False``. Specifically, when an immortal object is involved: * code that inspects the refcount will see a really, really large value * the new noop behavior may break code that: * depends specifically on the refcount to always increment or decrement (or have a specific value from ``Py_SET_REFCNT()``) * relies on any specific refcount value, other than 0 or 1 * directly manipulates the refcount to store extra information there * in 32-bit pre-3.11 `Stable ABI`_ extensions, objects may leak due to `Accidental Immortality`_ * such extensions may crash due to `Accidental De-Immortalizing`_ Again, those changes in behavior only apply to immortal objects, not most of the objects a user will access. Furthermore, users cannot mark an object as immortal so no user-created objects will ever have that changed behavior. Users that rely on any of the changing behavior for global (builtin) objects are already in trouble. So the overall impact should be small. Also note that code which checks for refleaks should keep working fine, unless it checks for hard-coded small values relative to some immortal object. The problems noticed by `Pyston`_ shouldn't apply here since we do not modify the refcount. See `Public Refcount Details`_ below for further discussion. Accidental Immortality '''''''''''''''''''''' Hypothetically, a non-immortal object could be incref'ed so much that it reaches the magic value needed to be considered immortal. That means it would accidentally never be cleaned up (by going back to 0). On 64-bit builds, this accidental scenario is so unlikely that we need not worry. Even if done deliberately by using ``Py_INCREF()`` in a tight loop and each iteration only took 1 CPU cycle, it would take 2^60 cycles (if the immortal bit were 2^60). At a fast 5 GHz that would still take nearly 250,000,000 seconds (over 2,500 days)! Also note that it is doubly unlikely to be a problem because it wouldn't matter until the refcount got back to 0 and the object was cleaned up. So any object that hit that magic "immortal" refcount value would have to be decref'ed that many times again before the change in behavior would be noticed. Again, the only realistic way that the magic refcount would be reached (and then reversed) is if it were done deliberately. (Of course, the same thing could be done efficiently using ``Py_SET_REFCNT()`` though that would be even less of an accident.) At that point we don't consider it a concern of this proposal. On 32-bit builds it isn't so obvious. Let's say the magic refcount were 2^30. Using the same specs as above, it would take roughly 4 seconds to accidentally immortalize an object. Under reasonable conditions, it is still highly unlikely that an object be accidentally immortalized. It would have to meet these criteria: * targeting a non-immortal object (so not one of the high-use builtins) * the extension increfs without a corresponding decref (e.g. returns from a function or method) * no other code decrefs the object in the meantime Even at a much less frequent rate incref it would not take long to reach accidental immortality (on 32-bit). However, then it would have to run through the same number of (now noop-ing) decrefs before that one object would be effectively leaking. This is highly unlikely, especially because the calculations assume no decrefs. Furthermore, this isn't all that different from how such 32-bit extensions can already incref an object past 2^31 and turn the refcount negative. If that were an actual problem then we would have heard about it. Between all of the above cases, the proposal doesn't consider accidental immortality a problem. Stable ABI '''''''''' The implementation approach described in this PEP is compatible with extensions compiled to the stable ABI (with the exception of `Accidental Immortality`_ and `Accidental De-Immortalizing`_). Due to the nature of the stable ABI, unfortunately, such extensions use versions of ``Py_INCREF()``, etc. that directly modify the object's ``ob_refcnt`` field. This will invalidate all the performance benefits of immortal objects. However, we do ensure that immortal objects (mostly) stay immortal in that situation. We set the initial refcount of immortal objects to a value high above the magic refcount value, but one that still matches the high bit. Thus we can still identify such objects as immortal. (See `_Py_IMMORTAL_REFCNT`_.) At worst, objects in that situation would feel the effects described in the `Motivation`_ section. Even then the overall impact is unlikely to be significant. Accidental De-Immortalizing ''''''''''''''''''''''''''' 32-bit builds of older stable ABI extensions can take `Accidental Immortality`_ to the next level. Hypothetically, such an extension could incref an object to a value on the next highest bit above the magic refcount value. For example, if the magic value were 2^30 and the initial immortal refcount were thus 2^30 + 2^29 then it would take 2^29 increfs by the extension to reach a value of 2^31, making the object non-immortal. (Of course, a refcount that high would probably already cause a crash, regardless of immortal objects.) The more problematic case is where such a 32-bit stable ABI extension goes crazy decref'ing an already immortal object. Continuing with the above example, it would take 2^29 asymmetric decrefs to drop below the magic immortal refcount value. So an object like ``None`` could be made mortal and subject to decref. That still wouldn't be a problem until somehow the decrefs continue on that object until it reaches 0. For many immortal objects, like ``None``, the extension will crash the process if it tries to dealloc the object. For the other immortal objects, the dealloc might be okay. However, there will be runtime code expecting the formerly-immortal object to be around forever. That code will probably crash. Again, the likelihood of this happening is extremely small, even on 32-bit builds. It would require roughly a billion decrefs on that one object without a corresponding incref. The most likely scenario is the following: A "new" reference to ``None`` is returned by many functions and methods. Unlike with non-immortal objects, the 3.11 runtime will almost never incref ``None`` before giving it to the extension. However, the extension *will* decref it when done with it (unless it returns it). Each time that exchange happens with the one object, we get one step closer to a crash. How realistic is it that some form of that exchange (with a single object) will happen a billion times in the lifetime of a Python process on 32-bit? If it is a problem, how could it be addressed? As to how realistic, the answer isn't clear currently. However, the mitigation is simple enough that we can safely proceed under the assumption that it would be a problem. Here are some possible solutions (only needed on 32-bit): * periodically reset the refcount for immortal objects (only enable this if a stable ABI extension is imported?) * special-case immortal objects in tp_dealloc() for the relevant types (but not int, due to frequency?) * provide a runtime flag for disabling immortality Alternate Python Implementations -------------------------------- This proposal is CPython-specific. However, it does relate to the behavior of the C-API, which may affect other Python implementations. Consequently, the effect of changed behavior described in `Backward Compatibility`_ above also applies here (e.g. if another implementation is tightly coupled to specific refcount values, other than 0, or on exactly how refcounts change, then they may impacted). Security Implications --------------------- This feature has no known impact on security. Maintainability --------------- This is not a complex feature so it should not cause much mental overhead for maintainers. The basic implementation doesn't touch much code so it should have much impact on maintainability. There may be some extra complexity due to performance penalty mitigation. However, that should be limited to where we immortalize all objects post-init and that code will be in one place. Specification ============= The approach involves these fundamental changes: * add `_Py_IMMORTAL_REFCNT`_ (the magic value) to the internal C-API * update ``Py_INCREF()`` and ``Py_DECREF()`` to no-op for objects with the magic refcount (or its most significant bit) * do the same for any other API that modifies the refcount * stop modifying ``PyGC_Head`` for immortal GC objects ("containers") * ensure that all immortal objects are cleaned up during runtime finalization Then setting any object's refcount to ``_Py_IMMORTAL_REFCNT`` makes it immortal. (There are other minor, internal changes which are not described here.) In the following sub-sections we dive into the details. First we will cover some conceptual topics, followed by more concrete aspects like specific affected APIs. Public Refcount Details ----------------------- In `Backward Compatibility`_ we introduced possible ways that user code might be broken by the change in this proposal. Any contributing misunderstanding by users is likely due in large part to the names of the refcount-related API and to how the documentation explains those API (and refcounting in general). Between the names and the docs, we can clearly see answers to the following questions: * what behavior do users expect? * what guarantees do we make? * do we indicate how to interpret the refcount value they receive? * what are the use cases under which a user would set an object's refcount to a specific value? * are users setting the refcount of objects they did not create? As part of this proposal, we must make sure that users can clearly understand on which parts of the refcount behavior they can rely and which are considered implementation details. Specifically, they should use the existing public refcount-related API and the only refcount values with any meaning are 0 and 1. (Some code relies on 1 as an indicator that the object can be safely modified.) All other values are considered "not 0 or 1". This information will be clarified in the `documentation <Documentation_>`_. Arguably, the existing refcount-related API should be modified to reflect what we want users to expect. Something like the following: * ``Py_INCREF()`` -> ``Py_ACQUIRE_REF()`` (or only support ``Py_NewRef()``) * ``Py_DECREF()`` -> ``Py_RELEASE_REF()`` * ``Py_REFCNT()`` -> ``Py_HAS_REFS()`` * ``Py_SET_REFCNT()`` -> ``Py_RESET_REFS()`` and ``Py_SET_NO_REFS()`` However, such a change is not a part of this proposal. It is included here to demonstrate the tighter focus for user expectations that would benefit this change. Constraints ----------- * ensure that otherwise immutable objects can be truly immutable * minimize performance penalty for normal Python use cases * be careful when immortalizing objects that we don't actually expect to persist until runtime finalization. * be careful when immortalizing objects that are not otherwise immutable * ``__del__`` and weakrefs must continue working properly Regarding "truly" immutable objects, this PEP doesn't impact the effective immutability of any objects, other than the per-object runtime state (e.g. refcount). So whether or not some immortal object is truly (or even effectively) immutable can only be settled separately from this proposal. For example, str objects are generally considered immutable, but ``PyUnicodeObject`` holds some lazily cached data. This PEP has no influence on how that state affects str immutability. Immortal Mutable Objects ------------------------ Any object can be marked as immortal. We do not propose any restrictions or checks. However, in practice the value of making an object immortal relates to its mutability and depends on the likelihood it would be used for a sufficient portion of the application's lifetime. Marking a mutable object as immortal can make sense in some situations. Many of the use cases for immortal objects center on immutability, so that threads can safely and efficiently share such objects without locking. For this reason a mutable object, like a dict or list, would never be shared (and thus no immortality). However, immortality may be appropriate if there is sufficient guarantee that the normally mutable object won't actually be modified. On the other hand, some mutable objects will never be shared between threads (at least not without a lock like the GIL). In some cases it may be practical to make some of those immortal too. For example, ``sys.modules`` is a per-interpreter dict that we do not expect to ever get freed until the corresponding interpreter is finalized. By making it immortal, we no longer incur the extra overhead during incref/decref. We explore this idea further in the `mitigation`_ section below. Implicitly Immortal Objects --------------------------- If an immortal object holds a reference to a normal (mortal) object then that held object is effectively immortal. This is because that object's refcount can never reach 0 until the immortal object releases it. Examples: * containers like ``dict`` and ``list`` * objects that hold references internally like ``PyTypeObject.tp_subclasses`` * an object's type (held in ``ob_type``) Such held objects are thus implicitly immortal for as long as they are held. In practice, this should have no real consequences since it really isn't a change in behavior. The only difference is that the immortal object (holding the reference) doesn't ever get cleaned up. We do not propose that such implicitly immortal objects be changed in any way. They should not be explicitly marked as immortal just because they are held by an immortal object. That would provide no advantage over doing nothing. Un-Immortalizing Objects ------------------------ This proposal does not include any mechanism for taking an immortal object and returning it to a "normal" condition. Currently there is no need for such an ability. On top of that, the obvious approach is to simply set the refcount to a small value. However, at that point there is no way in knowing which value would be safe. Ideally we'd set it to the value that it would have been if it hadn't been made immortal. However, that value has long been lost. Hence the complexities involved make it less likely that an object could safely be un-immortalized, even if we had a good reason to do so. _Py_IMMORTAL_REFCNT ------------------- We will add two internal constants:: _Py_IMMORTAL_BIT - has the top-most available bit set (e.g. 2^62) _Py_IMMORTAL_REFCNT - has the two top-most available bits set The actual top-most bit depends on existing uses for refcount bits, e.g. the sign bit or some GC uses. We will use the highest bit possible after consideration of existing uses. The refcount for immortal objects will be set to ``_Py_IMMORTAL_REFCNT`` (meaning the value will be halfway between ``_Py_IMMORTAL_BIT`` and the value at the next highest bit). However, to check if an object is immortal we will compare (bitwise-and) its refcount against just ``_Py_IMMORTAL_BIT``. The difference means that an immortal object will still be considered immortal, even if somehow its refcount were modified (e.g. by an older stable ABI extension). Note that top two bits of the refcount are already reserved for other uses. That's why we are using the third top-most bit. Affected API ------------ API that will now ignore immortal objects: * (public) ``Py_INCREF()`` * (public) ``Py_DECREF()`` * (public) ``Py_SET_REFCNT()`` * (private) ``_Py_NewReference()`` API that exposes refcounts (unchanged but may now return large values): * (public) ``Py_REFCNT()`` * (public) ``sys.getrefcount()`` (Note that ``_Py_RefTotal`` and ``sys.gettotalrefcount()`` will not be affected.) Also, immortal objects will not participate in GC. Immortal Global Objects ----------------------- All runtime-global (builtin) objects will be made immortal. That includes the following: * singletons (``None``, ``True``, ``False``, ``Ellipsis``, ``NotImplemented``) * all static types (e.g. ``PyLong_Type``, ``PyExc_Exception``) * all static objects in ``_PyRuntimeState.global_objects`` (e.g. identifiers, small ints) The question of making them actually immutable (e.g. for per-interpreter GIL) is not in the scope of this PEP. Object Cleanup -------------- In order to clean up all immortal objects during runtime finalization, we must keep track of them. For GC objects ("containers") we'll leverage the GC's permanent generation by pushing all immortalized containers there. During runtime shutdown, the strategy will be to first let the runtime try to do its best effort of deallocating these instances normally. Most of the module deallocation will now be handled by ``pylifecycle.c:finalize_modules()`` which cleans up the remaining modules as best as we can. It will change which modules are available during ``__del__``, but that's already explicitly undefined behavior in the docs. Optionally, we could do some topological ordering to guarantee that user modules will be deallocated first before the stdlib modules. Finally, anything left over (if any) can be found through the permanent generation GC list which we can clear after ``finalize_modules()``. For non-container objects, the tracking approach will vary on a case-by-case basis. In nearly every case, each such object is directly accessible on the runtime state, e.g. in a ``_PyRuntimeState`` or ``PyInterpreterState`` field. We may need to add a tracking mechanism to the runtime state for a small number of objects. None of the cleanup will have a significant effect on performance. .. _mitigation: Performance Regression Mitigation --------------------------------- In the interest of clarity, here are some of the ways we are going to try to recover some of the lost `performance <Performance_>`_: * at the end of runtime init, mark all objects as immortal * drop refcount operations in code where we know the object is immortal (e.g. ``Py_RETURN_NONE``) * specialize for immortal objects in the eval loop (see `Pyston`_) Regarding that first point, we can apply the concept from `Immortal Mutable Objects`_ in the pursuit of getting back some of that 4% performance we lose with the naive implementation of immortal objects. At the end of runtime init we can mark *all* objects as immortal and avoid the extra cost in incref/decref. We only need to worry about immutability with objects that we plan on sharing between threads without a GIL. Note that none of this section is part of the proposal. The above is included here for clarity. Possible Changes ---------------- * mark every interned string as immortal * mark the "interned" dict as immortal if shared else share all interned strings * (Larry,MvL) mark all constants unmarshalled for a module as immortal * (Larry,MvL) allocate (immutable) immortal objects in their own memory page(s) Documentation ------------- The immortal objects behavior and API are internal, implementation details and will not be added to the documentation. However, we will update the documentation to make public guarantees about refcount behavior more clear. That includes, specifically: * ``Py_INCREF()`` - change "Increment the reference count for object o." to "Indicate taking a new reference to object o." * ``Py_DECREF()`` - change "Decrement the reference count for object o." to "Indicate no longer using a previously taken reference to object o." * similar for ``Py_XINCREF()``, ``Py_XDECREF()``, ``Py_NewRef()``, ``Py_XNewRef()``, ``Py_Clear()`` * ``Py_REFCNT()`` - add "The refcounts 0 and 1 have specific meanings and all others only mean code somewhere is using the object, regardless of the value. 0 means the object is not used and will be cleaned up. 1 means code holds exactly a single reference." * ``Py_SET_REFCNT()`` - refer to ``Py_REFCNT()`` about how values over 1 may be substituted with some over value We *may* also add a note about immortal objects to the following, to help reduce any surprise users may have with the change: * ``Py_SET_REFCNT()`` (a no-op for immortal objects) * ``Py_REFCNT()`` (value may be surprisingly large) * ``sys.getrefcount()`` (value may be surprisingly large) Other API that might benefit from such notes are currently undocumented. We wouldn't add such a note anywhere else (including for ``Py_INCREF()`` and ``Py_DECREF()``) since the feature is otherwise transparent to users. Reference Implementation ======================== The implementation is proposed on GitHub: https://github.com/python/cpython/pull/19474 Open Issues =========== * how realistic is the `Accidental De-Immortalizing`_ concern? References ========== .. _Pyston: https://mail.python.org/archives/list/python-dev@python.org/message/JLHRTBJ… Prior Art --------- * `Pyston`_ Discussions ----------- This was discussed in December 2021 on python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/7O3FUA52… * https://mail.python.org/archives/list/python-dev@python.org/thread/PNLBJBNI… Runtime Object State -------------------- Here is the internal state that the CPython runtime keeps for each Python object: * `PyObject.ob_refcnt`_: the object's `refcount <refcounting_>`_ * `_PyGC_Head <PyGC_Head>`_: (optional) the object's node in a list of `"GC" objects <refcounting_>`_ * `_PyObject_HEAD_EXTRA <PyObject_HEAD_EXTRA>`_: (optional) the object's node in the list of heap objects ``ob_refcnt`` is part of the memory allocated for every object. However, ``_PyObject_HEAD_EXTRA`` is allocated only if CPython was built with ``Py_TRACE_REFS`` defined. ``PyGC_Head`` is allocated only if the object's type has ``Py_TPFLAGS_HAVE_GC`` set. Typically this is only container types (e.g. ``list``). Also note that ``PyObject.ob_refcnt`` and ``_PyObject_HEAD_EXTRA`` are part of ``PyObject_HEAD``. .. _PyObject.ob_refcnt: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyGC_Head: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _PyObject_HEAD_EXTRA: https://github.com/python/cpython/blob/80a9ba537f1f1666a9e6c5eceef4683f8696… .. _refcounting: Reference Counting, with Cyclic Garbage Collection -------------------------------------------------- Garbage collection is a memory management feature of some programming languages. It means objects are cleaned up (e.g. memory freed) once they are no longer used. Refcounting is one approach to garbage collection. The language runtime tracks how many references are held to an object. When code takes ownership of a reference to an object or releases it, the runtime is notified and it increments or decrements the refcount accordingly. When the refcount reaches 0, the runtime cleans up the object. With CPython, code must explicitly take or release references using the C-API's ``Py_INCREF()`` and ``Py_DECREF()``. These macros happen to directly modify the object's refcount (unfortunately, since that causes ABI compatibility issues if we want to change our garbage collection scheme). Also, when an object is cleaned up in CPython, it also releases any references (and resources) it owns (before it's memory is freed). Sometimes objects may be involved in reference cycles, e.g. where object A holds a reference to object B and object B holds a reference to object A. Consequently, neither object would ever be cleaned up even if no other references were held (i.e. a memory leak). The most common objects involved in cycles are containers. CPython has dedicated machinery to deal with reference cycles, which we call the "cyclic garbage collector", or often just "garbage collector" or "GC". Don't let the name confuse you. It only deals with breaking reference cycles. See the docs for a more detailed explanation of refcounting and cyclic garbage collection: * https://docs.python.org/3.11/c-api/intro.html#reference-counts * https://docs.python.org/3.11/c-api/refcounting.html * https://docs.python.org/3.11/c-api/typeobj.html#c.PyObject.ob_refcnt * https://docs.python.org/3.11/c-api/gcsupport.html Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

3 6

SC accepted PEP 594: Removing dead batteries from the standard library
by Victor Stinner 14 Mar '22

14 Mar '22

Hi, Oh, the Steering Council accepted PEP 594 "Removing dead batteries from the standard library"! I just saw the announcement on Discourse. Congratulations Christian and Brett! This PEP, first proposed in 2019, wasn't an easy one. https://peps.python.org/pep-0594/ Gregory P. Smith's message on Discourse: """ On behalf of the Python Steering Council, We are accepting PEP-594 32 Removing dead batteries from the standard library. It removes a non-controversial set of very old unmaintained or obsolete libraries from the Python standard library. We expect this PEP to be a one time event, and for future deprecations to be handled differently. One thing we’d like to see happen while implementing it: Document the status of the modules being deprecated and removed and backport those deprecation updates to older CPython branch documentation (at least back to 3.9). That gets the notices in front of more people who may use the docs for their specific Python version. Particular care should also be taken during the pre-release cycles that remove deprecated modules. If it turns out the removal of a module proves to be a problem in practice despite the clear deprecation, deferring the removal of that module should be considered to avoid disruption. Doing a “mass cleanup” of long obsolete modules is a sign that we as a project have been ignoring rather than maintaining parts of the standard library, or not doing so with the diligence being in the standard library implies they deserve. Resolving ongoing discussions around how we define the stdlib for the long term does not block this PEP. It seems worthwhile for us to conduct regular reviews of the contents of the stdlib every few releases so we can avoid accumulating such a large pile of dead batteries, but this is outside the scope of this particular PEP. – Greg for the PSC """ https://discuss.python.org/t/pep-594-take-2-removing-dead-batteries-from-th… Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 0

An unambiguous way of initializing an empty dictionary and set
by joao.p.f.batista.97＠gmail.com 14 Mar '22

14 Mar '22

Currently: l = [] # new empty list t = () # new empty tuple s = set() # new empty set (no clean and consistent way of initializing regarding the others) <<< d = {} # new empty dictionary Possible solution: s = {} # new empty set d = {:} # new empty dictionary (the ":" is a reference to key-value pairs) Current workaround at least for consistency: l = list() # new empty list t = tuple() # new empty tuple s = set() # new empty set d = dict() # new empty dictionary However, it doesn't feel right to not be able to initialize an empty set as cleanly and consistently as lists, tuples and dictionaries in both forms.

5 6

Re: cpython: simplify and rewrite the zipimport part of 702009f3c0b1 a bit
by Tommie Cadieux 14 Mar '22

14 Mar '22

1 0

Supported make implementations
by Lincoln Auster 12 Mar '22

12 Mar '22

Hi, When attempting to build and run doctests for cpython on FreeBSD, I got a vague error from the default BSD make. gmake, of course, ran fine. Is GNU make the only supported make implementation? If so, did I miss that in the devguide, or should that be in there somewhere? -- Lincoln Auster They/Them

2 1