On Thu, Sep 7, 2017 at 8:11 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
First of all, thanks for the feedback and encouragement! Responses in-line below.
I hope it's helpful! More responses in-line as well.
On Thu, Sep 7, 2017 at 3:48 PM, Nathaniel Smith <njs@pobox.com> wrote:
My concern about this is the same as it was last time -- the work looks neat, but right now, almost no-one uses subinterpreters (basically it's Jep and mod_wsgi and that's it?), and therefore many packages get away with ignoring subinterpreters.
My concern is that this is a chicken-and-egg problem. The situation won't improve until subinterpreters are more readily available.
Okay, but you're assuming that "more libraries work well with subinterpreters" is in fact an improvement. I'm asking you to convince me of that :-). Are there people saying "oh, if only subinterpreters had a Python API and less weird interactions with C extensions, I could do <something awesome>"? So far they haven't exactly taken the world by storm...
Numpy is the one I'm most familiar with: when we get subinterpreter bugs we close them wontfix, because supporting subinterpreters properly would require non-trivial auditing, add overhead for non-subinterpreter use cases, and benefit a tiny tiny fraction of our users.
The main problem of which I'm aware is C globals in libraries and extension modules. PEPs 489 and 3121 are meant to help but I know that there is at least one major situation which is still a blocker for multi-interpreter-safe module state. Other than C globals, is there some other issue?
That's the main one I'm aware of, yeah, though I haven't looked into it closely.
If we add a friendly python-level API like this, then we're committing to this being a part of Python for the long term and encouraging people to use it, which puts pressure on downstream packages to do that work... but it's still not clear whether any benefits will actually materialize.
I'm fine with Nick's idea about making this a "provisional" module. Would that be enough to ease your concern here?
Potentially, yeah -- basically I'm fine with anything that doesn't end up looking like python-dev telling everyone "subinterpreters are the future! go forth and yell at any devs who don't support them!". What do you think the criteria for graduating to non-provisional status should be, in this case? [snip]
So the only case I can see where I'd expect subinterpreters to make communication dramatically more efficient is if you have a "deeply immutable" type [snip] However, it seems impossible to support user-defined deeply-immutable types in Python: [snip]
I agree that it is currently not an option. That is part of the exercise. There are a number of possible solutions to explore once we get to that point. However, this PEP isn't about that. I'm confident enough about the possibilities that I'm comfortable with moving forward here.
I guess I would be much more confident in the possibilities here if you could give: - some hand-wavy sketch for how subinterpreter A could call a function that as originally defined in subinterpreter B without the GIL, which seems like a precondition for sharing user-defined classes - some hand-wavy sketch for how refcounting will work for objects shared between multiple subinterpreters without the GIL, without majorly impacting single-thread performance (I actually forgot about this problem in my last email, because PyPy has already solved this part!) These are the two problems where I find it most difficult to have faith. [snip]
I hope so -- a really useful subinterpreter multi-core stor[y] would be awesome.
Agreed! Thanks for the encouragement. :)
Thanks for attempting such an ambitious project :-). -n -- Nathaniel J. Smith -- https://vorpus.org