On Thu, 25 May 2017 at 08:06 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 25 May 2017 at 13:30, Guido van Rossum <guido@python.org> wrote:
Hm... Curiously, I've heard a few people at PyCon mention they thought subinterpreters were broken and not useful (and they share the GIL anyways) and should be taken out.
Taking them out entirely would break mod_wsgi (and hence the use of Apache httpd as a Python application server), so I hope we don't consider going down that path :)
As far as the GIL goes, Eric has a few ideas around potentially getting to a tiered locking approach, where the GIL becomes a Read/Write lock shared across the interpreters, and there are separate subinterpreter locks to guard actual code execution. That becomes a lot more feasible in a subinterpreter model, since the eval loop and various other structures are already separate - the tiered locking would mainly need to account for management of "object ownership" that prevented multiple interpreters from accessing the same object at the same time.
However, I do think subinterpreters can be accurately characterised as fragile, especially in the presence of extension modules. I also think a large part of that fragility can be laid at the feet of them currently being incredibly difficult to test - while _testembed includes a rudimentary check [1] to make sure the subinterpreter machinery itself basically works, it doesn't do anything in the way of checking that the rest of the standard library actually does the right thing when run in a subinterpreter.
So I'm +1 for the idea of exposing a low-level CPython-specific _interpreters API in order to start building out a proper test suite for the capability, and to let folks interested in working with them do so without having to write a custom embedding application ala mod_wsgi.
However, I think it's still far too soon to be talking about defining a public supported API for them - while their use in mod_wsgi gives us assurance that they do mostly work in CPython, other implementations don't necessarily have anything comparable (even as a private implementation detail), and the kinds of software that folks run directly under mod_wsgi isn't necessarily reflective of the full extent of variation in the kinds of code that Python developers write in general.
I'm +1 on Nick's idea of the low-level, private API existing first to facilitate testing, but putting off any public API until we're sure we can make it function in a way we're happy with to more generally expose.