On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 21 Apr 2020 18:46:04 +0200
Petr Viktorin <encukou@gmail.com> wrote:
> On 2020-04-21 11:01, Antoine Pitrou wrote:
> > On Mon, 20 Apr 2020 19:21:21 -0600
> > Eric Snow <ericsnowcurrently@gmail.com> wrote: 
> >> Honest question: how many C extensions have process-global state that
> >> will cause problems under subinterpreters?  In other words, how many
> >> already break in mod_wsgi? 
> >
> > A slightly tricky question is what happens if a PyObject survives
> > longer than the subinterpreter that created it.
> >
> > For example, in PyArrow, we allow passing a Python buffer-like object
> > as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
> > around by C++ code for some time.  When the C++ buffer is destroyed,
> > Py_DECREF() is called on the Python object (I figure that we would have
> > to switch to the future interpreter-aware PyGILState API -- when will
> > it be available?).   

In this scenario given that the PyObject owning the buffer passed to C++ by PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release anything who's refcount never made it to 0.  Which also implies leaving enough of the interpreter state around so that Py_DECREF could be called and trigger the free().

Basically (sub)interpreter finalization should have the right to return failure.

Everything ever allocated by an interpreter holds a reference (implied today) to its associated interpreter state.

> > But what happens if the interpreter which created
> > the Python object is already destroyed at this point? 
> That's a good question. What happens today if someone calls Py_Finalize
> while the buffer is still around?
> I don't think you need to treat *sub*interpreters specially.

The difference is that subinterpreters may have a shorter lifetime than
the main interpreter.  Also, we're currently using a global memory
allocator, so simple operations may work until the process dies.



Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UWWO3RS3UOYCZXVFORGTJAEXXOCUQQXD/
Code of Conduct: http://python.org/psf/codeofconduct/