On 09. 03. 22 4:38, Eric Snow wrote:
I'd really appreciate feedback on this new PEP about making the GIL per-interpreter.
Yay! Thank you! This PEP definitely makes per-interpreter GIL sound possible :)
The PEP targets 3.11, but we'll see if that is too close. I don't mind waiting one more release, though I'd prefer 3.11 (obviously). Regardless, I have no intention of rushing this through at the expense of cutting corners. Hence, we'll see how it goes.
The PEP text is included inline below. Thanks!
-eric
===================================================
PEP: 684 Title: A Per-Interpreter GIL Author: Eric Snow <ericsnowcurrently@gmail.com> Discussions-To: python-dev@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst
This iteration of the PEP should also have `Requires: 683` (Immortal Objects). [...]
Motivation ==========
The fundamental problem we're solving here is a lack of true multi-core parallelism (for Python code) in the CPython runtime. The GIL is the cause. While it usually isn't a problem in practice, at the very least it makes Python's multi-core story murky, which makes the GIL a consistent distraction.
Isolated interpreters are also an effective mechanism to support certain concurrency models. :pep:`554` discusses this in more detail.
Indirect Benefits -----------------
Most of the effort needed for a per-interpreter GIL has benefits that make those tasks worth doing anyway:
* makes multiple-interpreter behavior more reliable * has led to fixes for long-standing runtime bugs that otherwise hadn't been prioritized > * has been exposing (and inspiring fixes for) previously unknown
runtime bugs
* has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) * has driven cleaner and more complete runtime finalization * led to structural layering of the C-API (e.g. ``Include/internal``) * also see `Benefits to Consolidation`_ below
Do you want to dig up some bpo examples, to make these more convincing to the casual reader?
Furthermore, much of that work benefits other CPython-related projects:
* performance improvements ("faster-cpython") * pre-fork application deployment (e.g. Instagram)
Maybe say “e.g. with Instagram's Cinder” – both the household name and the project you can link to?
* extension module isolation (see :pep:`630`, etc.) * embedding CPython
A lot of these points are duplicated in "Benefits to Consolidation" list below, maybe there'd be, ehm, benefits to consolidating them? [...]
PEP 554 -------
Please spell out "PEP 554 (Multiple Interpreters in the Stdlib)", for people who don't remember the magic numbers but want to skim the table of contents.
:pep:`554` is strictly about providing a minimal stdlib module to give users access to multiple interpreters from Python code. In fact, it specifically avoids proposing any changes related to the GIL. Consider, however, that users of that module would benefit from a per-interpreter GIL, which makes PEP 554 more appealing.
Rationale =========
During initial investigations in 2014, a variety of possible solutions for multi-core Python were explored, but each had its drawbacks without simple solutions:
* the existing practice of releasing the GIL in extension modules * doesn't help with Python code * other Python implementations (e.g. Jython, IronPython) * CPython dominates the community * remove the GIL (e.g. gilectomy, "no-gil") * too much technical risk (at the time) * Trent Nelson's "PyParallel" project * incomplete; Windows-only at the time * ``multiprocessing``
* too much work to make it effective enough; high penalties in some situations (at large scale, Windows)
* other parallelism tools (e.g. dask, ray, MPI) * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) * this can only end in tears
This list doesn't render correctly in ReST, you need blank lines everywhere. There are more cases like this below. [...]> Per-Interpreter State
---------------------
The following runtime state will be moved to ``PyInterpreterState``:
* all global objects that are not safely shareable (fully immutable) * the GIL * mutable, currently protected by the GIL
Spelling out “mutable state” in these lists would make this clearer, since “state” isn't elided from all the points.
* mutable, currently protected by some other per-interpreter lock * mutable, may be used independently in different interpreters
This includes extension modules (with multi-phase init), right?
* all other mutable (or effectively mutable) state not otherwise excluded below
Furthermore, a number of parts of the global state have already been moved to the interpreter, such as GC, warnings, and atexit hooks.
The following state will not be moved:
* global objects that are safely shareable, if any * immutable, often ``const`` * treated as immutable
Do you have an example for this?
* related to CPython's ``main()`` execution * related to the REPL
Would “only used by” work instead of “related to”?
* set during runtime init, then treated as immutable
`main()`, REPL and runtime init look like special cases of functionality that only runs in one interpreter. If it's so, maybe generalize this? [...]
C-API -----
The following private API will be made public:
* ``_PyInterpreterConfig`` * ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``)
Since the API is not documented (and _PyInterpreterConfig is not even in main yet!), it would be good to sketch out the docs (intended behavior) here.
The following fields will be added to ``PyInterpreterConfig``:
* ``own_gil`` - (bool) create a new interpreter lock (instead of sharing with the main interpreter)
As a user of the API, what should I consider when setting this flag? Would the GIL be shared with the *parent* interpreter or the main one? What are the restrictions/implications of this flag?
* ``strict_extensions`` - fail import in this interpreter for incompatible extensions (see `Restricting Extension Modules`_)
I'm not sure about including a workaround flag in the structure. Since the Python API will get a context manager for this, maybe the C API should get a function to set/reset it instead of this flag?
Restricting Extension Modules -----------------------------
Extension modules have many of the same problems as the runtime when state is stored in global variables. :pep:`630` covers all the details of what extensions must do to support isolation, and thus safely run in multiple interpreters at once. This includes dealing with their globals.
Extension modules that do not implement isolation will only run in the main interpreter. In all other interpreters, the import will raise ``ImportError``. This will be done through ``importlib._bootstrap_external.ExtensionFileLoader``.
“Main interpreter” should be defined. (Or maybe the term should be avoided instead -- always having to spell out “interpreter started by Py_Initialize rather than Py_NewInterpreter” might push us toward finding ways to avoid the special case...)
We will work with popular extensions to help them support use in multiple interpreters. This may involve adding to CPython's public C-API, which we will address on a case-by-case basis.
Extension Module Compatibility ''''''''''''''''''''''''''''''
As noted in `Extension Modules`_, many extensions work fine in multiple interpreters without needing any changes. The import system will still fail if such a module doesn't explicitly indicate support. At first, not many extension modules will, so this is a potential source of frustration.
We will address this by adding a context manager to temporarily disable the check on multiple interpreter support: ``importlib.util.allow_all_extensions()``. >
I'd prefer a more dangerous-sounding name, to guide code readers (and autocomplete users) toward checking the warning in the docs. [...]
Extension Modules '''''''''''''''''
Currently the most common usage of Python, by far, is with the main interpreter running by itself. This proposal has zero impact on extension modules in that scenario. Likewise, for better or worse, there is no change in behavior under multiple interpreters created using the existing ``Py_NewInterpreter()``.
Keep in mind that some extensions already break when used in multiple interpreters, due to keeping module state in global variables. They may crash or, worse, experience inconsistent behavior. That was part of the motivation for :pep:`630` and friends, so this is not a new situation nor a consequence of this proposal.
In contrast, when the `proposed API <proposed capi_>`_ is used to create multiple interpreters, the default behavior will change for some extensions. In that case, importing an extension will fail (outside the main interpreter) if it doesn't indicate support for multiple interpreters. For extensions that already break in multiple interpreters, this will be an improvement.
Now we get to the break in compatibility mentioned above. Some extensions are safe under multiple interpreters, even though they haven't indicated that. Unfortunately, there is no reliable way for the import system to infer that such an extension is safe, so importing them will still fail. This case is addressed in `Extension Module Compatibility`_ below.
Extensions that use multi-phase init should already be compatible with multiple interpreters. Multi-phase init itself is the flag that indicates this. But they might not be compatible with *per-interpreter GIL*. I don't like how that's conflated with multiple interpreters here. For example, extension modules can currently support multiple interpreters, but rely on the GIL to protect calls to a non-threadsafe library, access shared memory, etc. As an example, the PEP 630 “opt-out” is not thread-safe. It seems to me that there should be a separate flag (slot) to indicate support for per-interpreter GIL, and the `strict_extensions` bit should work with that. [...]
How to Teach This =================
This is an advanced feature for users of the C-API. There is no expectation that this will be taught. Oh, I'm afraid this will need some docs related to making sure an extension is compatible with per-interpreter GIL. I'd rather not repeat my mistake of hand-wavingly noting "All modules created using multi-phase initialization are expected to support sub-interpreters" in the docs, and only writing PEP 630 much later.
[...]
References ==========
Related:
* :pep:`384` * :pep:`432` * :pep:`489` * :pep:`554` * :pep:`573` * :pep:`587` * :pep:`630` * :pep:`683` * :pep:`3121`
Please write out the titles here.