On 21 Apr 2020, at 16:58, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Thanks for explaining that, Ronald. It sounds like a lot of the effort would relate to making classes work. I have some comments in-line below.
-eric
On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 21 Apr 2020, at 03:21, Eric Snow <ericsnowcurrently@gmail.com> wrote: Honest question: how many C extensions have process-global state that will cause problems under subinterpreters? In other words, how many already break in mod_wsgi?
Fully supporting sub-interpreters in PyObjC will likely be a lot of work, mostly due to being able to subclass Objective-C classes from Python. With sub- interpreters a Python script in an interpreter could see an Objective-C class in a different sub-interpreter. The current PyObjC architecture assumes that there’s exactly one (sub-)interpreter, that’s probably fixable but is far from trivial.
Are the Objective-C classes immutable? Are the wrappers stateful at all? Without context I'm not clear on how you would be impacted by operation under subinterpreters (i.e. PEP 554), but it sounds like the classes do have global state that is in fact interpreter-specific. I expect you would also be impacted by subinterpreters not sharing the GIL but that is a separate matter (see below).
My mail left out some important information, sorry about that. PyObjC is a two-way bridge between Python and Objective-C. One half of this is that is bridging Objective-C classes (and instances) to Python. This is fairly straightforward, although the proxy objects are not static and can have methods defined in Python (helper methods that make the Objective-C classes nicer to use from Python, for example to define methods that make it possible to use an NSDictionary as if it were a regular Python dict). The other half is that it is possible to implement Objective-C classes in Python: class MyClass (Cocoa.NSObject): def anAction_(self, sender): … This defines a Python classes named “MyClass”, but also an Objective-C class of the same name that forwards Objective-C calls to Python. The implementation for this uses PyGILState_Ensure, which AFAIK is not yet useable with sub-interpreters. PyObjC also has Objective-C proxy classes for generic Python objects, making it possible to pass a normal Python dictionary to an Objective-C API that expects an NSDictionary instance. Things get interesting when combining the two with sub-interpreters: With the current implementation the Objective-C world would be a channel for passing “live” Python objects between sub-interpreters. The translation tables for looking up existing proxies (mapping from Python to Objective-C and vice versa) are currently singletons. This is probably fixable with another level of administration, by keeping track of the sub-interpreter that owns a Python object I could ensure that Python objects owned by a different sub-interpreter are proxied like any other Objective-C object which would close this loophole. That would require significant changes to a code base that’s already fairly complex, but should be fairly straightforward.
Regardless, I expect there are others in a similar situation. It would be good to understand your use case and help with a solution. Is there a specific example you can point to of code that would be problematic under subinterpreters?
With the current API it might not even be possible to add sub-interpreter support
What additional API would be needed?
See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of time thinking about this though, I might find other issues when I try to support sub-interpreters.
(although I write this without having read the PEP).
Currently PEP 554 does not talk about how to make extension modules compatible with subinterpreters. That may be worth doing, though it would definitely have to happen in the docs and (to an extent) the 3.9 "What's New" page. There is already some discussion on what should be in those docs (see https://github.com/ericsnowcurrently/multi-core-python/issues/53).
Note that, until the GIL becomes per-interpreter, sharing objects isn't a problem. We were not planning on having a PEP for the stop-sharing-the-GIL effort, but I'm starting to think that it may be worth it, to cover the impact on extension modules (e.g. mitigations).
So if you leave out the complications due to not sharing the GIL, the main problem extension authors face with subinterpreters (exposed by PEP 554) is when their module has process-global state that breaks under subinterpreters. From your description above, it sounds like you may be in that situation.
As far as I understand proper support for subinterpreters also requires moving away from static type definitions to avoid sharing objects between interpreters (that is, use the PyType_FromSpec to build types).
Correct, though that is not technically a problem until we stop sharing the GIL.
Right. But a major selling point of sub-interpreters is that this provide a way forward towards having multiple Python threads that don’t share a GIL. IMHO it would be better to first work out what’s needed to get there, and in particular what changes are needed in extensions. Otherwise extensions may have to be changed multiple times.
At first glance this API does not support everything I do in PyObjC (fun with metaclasses, in C code).
What specific additions/changes would you need?
At least: - A variant of PyGILState_Ensure that supports sub-interpreters - Defining subclasses of built-in types using PyType_FromSpec, in particular a subclass of “type”. BTW. In my first mail I mentioned I don’t have a use cases for subinterpreters. I might have a limited use case in the PyObjC domain: implementing plugin bundles for Objective-C applications in Python. These currently share the same interpreter, which can cause problems. Subinterpreters could be helpful there to isolate code, but that would require having an API that conditionally initialises the Python runtime (similar to PyGILState_Ensure, but for the runtime itself). This wouldn’t fix all problems because you can’t have two different python versions in one proces, but would be better than the status quo. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/