
On Thu, Sep 7, 2017 at 11:26 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Hi all,
As part of the multi-core work I'm proposing the addition of the "interpreters" module to the stdlib. This will expose the existing subinterpreters C-API to Python code. I've purposefully kept the API simple. Please let me know what you think.
My concern about this is the same as it was last time -- the work looks neat, but right now, almost no-one uses subinterpreters (basically it's Jep and mod_wsgi and that's it?), and therefore many packages get away with ignoring subinterpreters. Numpy is the one I'm most familiar with: when we get subinterpreter bugs we close them wontfix, because supporting subinterpreters properly would require non-trivial auditing, add overhead for non-subinterpreter use cases, and benefit a tiny tiny fraction of our users. If we add a friendly python-level API like this, then we're committing to this being a part of Python for the long term and encouraging people to use it, which puts pressure on downstream packages to do that work... but it's still not clear whether any benefits will actually materialize. I've actually argued with the PyPy devs to try to convince them to add subinterpreter support as part of their experiments with GIL-removal, because I think the semantics would genuinely be nicer to work with than raw threads, but they're convinced that it's impossible to make this work. Or more precisely, they think you could make it work in theory, but that it would be impossible to make it meaningfully more efficient than using multiple processes. I want them to be wrong, but I have to admit I can't see a way to make it work either... If this is being justified by the multicore use case, and specifically by the theory that having two interpreters in the same process will allow for more efficient communication than two interpreters in two different processes, then... why should we believe that that's actually possible? I want your project to succeed, but if it's going to fail then it seems better if it fails before we commit to exposing new APIs. -n -- Nathaniel J. Smith -- https://vorpus.org