[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

Brett Cannon brett at python.org
Thu May 25 14:55:50 EDT 2017

On Thu, 25 May 2017 at 08:06 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 25 May 2017 at 13:30, Guido van Rossum <guido at python.org> wrote:
> > Hm... Curiously, I've heard a few people at PyCon mention they thought
> > subinterpreters were broken and not useful (and they share the GIL
> anyways)
> > and should be taken out.
> Taking them out entirely would break mod_wsgi (and hence the use of
> Apache httpd as a Python application server), so I hope we don't
> consider going down that path :)
> As far as the GIL goes, Eric has a few ideas around potentially
> getting to a tiered locking approach, where the GIL becomes a
> Read/Write lock shared across the interpreters, and there are separate
> subinterpreter locks to guard actual code execution. That becomes a
> lot more feasible in a subinterpreter model, since the eval loop and
> various other structures are already separate - the tiered locking
> would mainly need to account for management of "object ownership" that
> prevented multiple interpreters from accessing the same object at the
> same time.
> However, I do think subinterpreters can be accurately characterised as
> fragile, especially in the presence of extension modules. I also think
> a large part of that fragility can be laid at the feet of them
> currently being incredibly difficult to test - while _testembed
> includes a rudimentary check [1] to make sure the subinterpreter
> machinery itself basically works, it doesn't do anything in the way of
> checking that the rest of the standard library actually does the right
> thing when run in a subinterpreter.
> So I'm +1 for the idea of exposing a low-level CPython-specific
> _interpreters API in order to start building out a proper test suite
> for the capability, and to let folks interested in working with them
> do so without having to write a custom embedding application ala
> mod_wsgi.
> However, I think it's still far too soon to be talking about defining
> a public supported API for them - while their use in mod_wsgi gives us
> assurance that they do mostly work in CPython, other implementations
> don't necessarily have anything comparable (even as a private
> implementation detail), and the kinds of software that folks run
> directly under mod_wsgi isn't necessarily reflective of the full
> extent of variation in the kinds of code that Python developers write
> in general.

I'm +1 on Nick's idea of the low-level, private API existing first to
facilitate testing, but putting off any public API until we're sure we can
make it function in a way we're happy with to more generally expose.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170525/ee12b45f/attachment.html>

More information about the Python-ideas mailing list