On Wed, Apr 22, 2020 at 2:43 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
My mail left out some important information, sorry about that.
No worries. :)
PyObjC is a two-way bridge between Python and Objective-C. One half of this is that is bridging Objective-C classes (and instances) to Python. This is fairly straightforward, although the proxy objects are not static and can have methods defined in Python (helper methods that make the Objective-C classes nicer to use from Python, for example to define methods that make it possible to use an NSDictionary as if it were a regular Python dict).
Cool. (also fairly straightforward!)
The other half is that it is possible to implement Objective-C classes in Python:
class MyClass (Cocoa.NSObject): def anAction_(self, sender): …
This defines a Python classes named “MyClass”, but also an Objective-C class of the same name that forwards Objective-C calls to Python.
Even cooler! :)
The implementation for this uses PyGILState_Ensure, which AFAIK is not yet useable with sub-interpreters.
That is correct. It is one of the few major subinterpreter bugs/"bugs" remaining to be addressed in the CPython code. IIRC, there were several proposed solutions (between 2 BPO issues) that would fix it but we got distracted before the matter was settled.
PyObjC also has Objective-C proxy classes for generic Python objects, making it possible to pass a normal Python dictionary to an Objective-C API that expects an NSDictionary instance.
Also super cool. How similar is this to Jython and IronPython?
Things get interesting when combining the two with sub-interpreters: With the current implementation the Objective-C world would be a channel for passing “live” Python objects between sub-interpreters.
+1
The translation tables for looking up existing proxies (mapping from Python to Objective-C and vice versa) are currently singletons.
This is probably fixable with another level of administration, by keeping track of the sub-interpreter that owns a Python object I could ensure that Python objects owned by a different sub-interpreter are proxied like any other Objective-C object which would close this loophole. That would require significant changes to a code base that’s already fairly complex, but should be fairly straightforward.
Do you think there are any additions we could make to the C-API (more than have been done recently, e.g. PEP 573) that would make this easier. From what I understand, this pattern of a cache/table of global Python objects is a relatively common one. So anything we can do to help transition these to per-interpreter would be broadly beneficial. Ideally it would be done in the least intrusive way possible, reducing churn and touch points. (e.g. a macro to convert existing tables,etc. + an init func to call during module init.) Also, FWIW, I've been thinking about possible approaches where the first/main interpreter uses the existing static types, etc. and further subinterpreters use a heap type (etc.) derived mostly automatically from the static one. It's been on my mind because this is one of the last major hurdles to clear in the CPython code before we can make the GIL per-interpreter.
What additional API would be needed?
See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of time thinking about this though, I might find other issues when I try to support sub-interpreters.
Any feedback on this would be useful.
As far as I understand proper support for subinterpreters also requires moving away from static type definitions to avoid sharing objects between interpreters (that is, use the PyType_FromSpec to build types).
Correct, though that is not technically a problem until we stop sharing the GIL.
Right. But a major selling point of sub-interpreters is that this provide a way forward towards having multiple Python threads that don’t share a GIL.
IMHO it would be better to first work out what’s needed to get there, and in particular what changes are needed in extensions. Otherwise extensions may have to be changed multiple times.
Yeah, I see what you're saying. It's been a hard balance to strike. There are really 2 things that have to be done: move all global state to per-interpreter and deal with static types (etc.). Do you think both will require significant work in the community? My instinct, partly informed by my work in CPython along these lines, is that the former is more work and sometimes trickier. The latter is fairly straightforward and much more of an opportunity for automatic approaches.
At first glance this API does not support everything I do in PyObjC (fun with metaclasses, in C code).
What specific additions/changes would you need?
At least:
- A variant of PyGILState_Ensure that supports sub-interpreters - Defining subclasses of built-in types using PyType_FromSpec, in particular a subclass of “type”.
Thanks!
BTW. In my first mail I mentioned I don’t have a use cases for subinterpreters. I might have a limited use case in the PyObjC domain: implementing plugin bundles for Objective-C applications in Python. These currently share the same interpreter, which can cause problems. Subinterpreters could be helpful there to isolate code, but that would require having an API that conditionally initialises the Python runtime (similar to PyGILState_Ensure, but for the runtime itself).
Ah, interesting.
This wouldn’t fix all problems because you can’t have two different python versions in one proces, but would be better than the status quo.
FWIW, I've been involved in past discussions about the idea of supporting multiple Python runtimes (including difference versions) in the same process. :) On top of that, I know someone who demonstrated to me how to use dlmopen() to do something like this right now. :) -eric