
On Wed, May 6, 2020 at 2:25 PM Jeff Allen <ja.py@farowl.co.uk> wrote:
Many thanks for working on this so carefully for so long. I'm happy to see the per-interpreter GIL will now be studied fully before final commitment to subinterpreters in the stdlib. I would have chipped in in those terms to the review, but others succesfully argued for "provisional" inclusion, and I was content with that.
No problem. :)
My reason for worrying about this is that, while the C-API has been there for some time, it has not had heavy use in taxing cases AFAIK, and I think there is room for it to be incorrect. I am thinking more about Jython than CPython, but ideally they are the same structures. When I put the structures to taxing use cases on paper, they don't seem quite to work. Jython has been used in environments with thread-pools, concurrency, and multiple interpreters, and this aspect has had to be "fixed" several times.
That insight would be super helpful and much appreciated. :) Is that all on the docs you've linked?
My use cases include sharing objects between interpreters, which I know the PEP doesn't. The C-API docs acknowledge that object sharing can't be prevented, but do their best to discourage it because of the hazards around allocation. Trouble is, I think it can happen unawares. The fact that Java takes on lifecycle management suggests it shouldn't be a fundamental problem in Jython. I know from other discussion it's where many would like to end up, even in CPython.
Yeah, for now we will strictly disallow sharing actual objects between interpreters in Python Code. It would be an interesting project to try loosening that at some point (especially with immutable type), but we're going to start from the safer position. We have no plans to add any similar restrictions to the C-API, where by you're typically much more free to shoot your own foot. :)
This is all theory: I don't have even a model implementation, so I won't pontificate. However, I do have pictures, without which I find it impossible to think about this subject. I couldn't find your pictures, so share mine here (WiP):
https://the-very-slow-jython-project.readthedocs.io/en/latest/architecture/i...
I would be interested in how you solve the problem of finding the current interpreter, discussed in the article. My preferred answer is:
https://the-very-slow-jython-project.readthedocs.io/en/latest/architecture/i...
That's the API change I think is needed. It might not have a visible effect on the PEP, but it's worth bearing in mind the risk of exposing a thing you might shortly find you want to change.
This is great stuff, Jeff! Thanks for sharing it. I was able to skim through but don't have time to dig in at the moment. I'll reply in detail as soon as I can. In the meantime, the implementation of PEP 554 exposes a single part of PyInterpreterState: the ID (an int). The only other internal-ish info we expose is whether or not an interpreter (by ID) is currently running. The only functionality we provide is: create, destroy, and run_string(). -eric