[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

Jim Baker jim.baker at python.org
Thu May 25 13:33:52 EDT 2017


Something like these subinterpreters in CPython are used from Jython's Java
API. Like nearly all of Jython* this can be directly imported into using
Python code, as seen in tests using this feature:  https://github.com/

More on the API here: https://github.com/jythontools/jython/blob/
master/src/org/python/util/PythonInterpreter.java - note that is not even a
core API for Jython, it just happens to be widely used, including by the
launcher that wraps this API and calls itself the jython executable. So we
can readily refactor if we have something better, because right now it is
also problematic with respect to its lifecycle; what is the mapping to
threads; and how it interacts with class loaders and other resources,
especially during cleanup.

It would be helpful to coordinate this subinterpreter work; or at least to
cc jython-dev on such ideas your might have. Recently there have been some
rumblings of consensus that it's about time for Jython to really start work
on the 3.x implementation, targeting 3.6. But do be aware we are usually at
most 2 to 5 developers, working in our spare time. So everything takes much
longer than one would hope. I just hope we can finish 3.6 (or whatever)
before Python 4.0 arrives :)

*Excluding certain cases on core types where our bytecode rewriting makes
it a true challenge!

- Jim

On Thu, May 25, 2017 at 11:03 AM, Eric Snow <ericsnowcurrently at gmail.com>

> On Wed, May 24, 2017 at 8:30 PM, Guido van Rossum <guido at python.org>
> wrote:
> > Hm... Curiously, I've heard a few people at PyCon
> I'd love to get in touch with them and discuss the situation.  I've
> spoken with Graham Dumpleton on several occasions about
> subinterpreters and what needs to be fixed.
> > mention they thought subinterpreters were broken
> There are a number of related long-standing bugs plus a few that I
> created in the last year or two.  I'm motivated to get these resolved
> so that the multi-core Python project can take full advantage of
> subinterpreters without worry.
> As well, there are known limitations to using extension modules in
> subinterpreters.  However, only extension modules that rely on process
> globals (rather than leveraging PEP 384, etc.) are affected, and we
> can control for that more carefully using the protocol introduced by
> PEP 489.
> There isn't anything broken about the concept or design of
> subinterpreters in CPython that I'm aware of.
> > and not useful (and they share the GIL anyways)
> I'll argue that their usefulness has been limited by lack of exposure
> in the stdlib. :)  Furthermore, I'm finding them extremely useful as
> the vehicle for the multi-core Python project.
> > and should be taken out. So we should at least have clarity on which
> > direction we want to take...
> I'd definitely appreciate a firm commitment that they are not getting
> removed as I don't want to spend a ton of time on the project just to
> have the effort made irrelevant. :)  Also, I'd be surprised if there
> were sufficient merit to removing support for subinterpreters, since
> there is very little machinery just for that feature.  Instead, it
> leverages components of CPython that are there for other valid
> reasons.  So I do not consider subinterpreters to currently add any
> significant burden to maintenance or development of the code base.
> Regardless, exposing the low-level _subinterpreters module should help
> us iron out bugs and API, as Nick pointed out.
> -eric
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170525/96fbb9b6/attachment.html>

More information about the Python-ideas mailing list