[New-bugs-announce] [issue40572] Support basic asynchronous cross-interpreter operations.

[Eric Snow]

New submission from Eric Snow <ericsnowcurrently at gmail.com>:

(This is a continuation of the work from bpo-33608.  That issue ended up with a lot of baggage and clutter (due to problems that have since been resolved), so we closed it.  This issue is where we're picking it up fresh.)

When two interpreters are cooperating, there are sometimes operations that one of them needs the other to perform.  Thus far this is limited to mutation or release of data/objects owned by that "other" interpreter.  We need safe, reliable public C-API to facilitate such operations.

All the necessary machinery already exists ("pending calls"), exposed in the internal C-API: _Py_AddPendingCall().  (There is a public Py_AddPendingCall(), but it should be removed.)  That API adds a function to a per-interpreter queue, from which it is executed later (asynchronously) by the ceval loop of the interpreter's main thread.  The challenge is that the repercussions of such async operations within the eval loop are not fully explored.  We have some confidence because this is the same machinery used to handle signals.  However, it's better avoid as much complexity in those async operations as possible.  That is why we don't just expose `_Py_AddPendingCall()` in the public C-API.

Instead the plan is to add a small number of public C-API functions that are each specific to a focused use case, ideally with with little/no chance of errors or other side effects.  Initially we will target `Py_DECREF()`, `PyMem_Free()`, and `PyBuffer_Release()`.  If we need more then we can assess those needs later.

----------
assignee: eric.snow
components: C API
messages: 368476
nosy: eric.snow, vstinner
priority: normal
severity: normal
stage: needs patch
status: open
title: Support basic asynchronous cross-interpreter operations.
type: enhancement
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40572>
_______________________________________