On 29 Apr 2020, at 03:50, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Wed, Apr 22, 2020 at 2:43 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
My mail left out some important information, sorry about that.
No worries. :)
PyObjC is a two-way bridge between Python and Objective-C. One half of this is that is bridging Objective-C classes (and instances) to Python. This is fairly straightforward, although the proxy objects are not static and can have methods defined in Python (helper methods that make the Objective-C classes nicer to use from Python, for example to define methods that make it possible to use an NSDictionary as if it were a regular Python dict).
Cool. (also fairly straightforward!)
Well… Except that the proxy classes are created dynamically and the list of methods is updated dynamically as well both for performance reasons and because ObjC classes can be changed at runtime (similar to how you can add methods to Python classes). But in the end this part is fairly straightforward and comparable to something like gobject introspection in the glib/gtk bindings. And every Cocoa class is proxied with a regular class and a metaclass (with parallel class hierarchies). That’s needed to mirror Objective-C behaviour, where class- and instance methods have separate namespaces and some classes have class and instance methods with the same name.
The other half is that it is possible to implement Objective-C classes in Python:
class MyClass (Cocoa.NSObject): def anAction_(self, sender): …
This defines a Python classes named “MyClass”, but also an Objective-C class of the same name that forwards Objective-C calls to Python.
Even cooler! :)
The implementation for this uses PyGILState_Ensure, which AFAIK is not yet useable with sub-interpreters.
That is correct. It is one of the few major subinterpreter bugs/"bugs" remaining to be addressed in the CPython code. IIRC, there were several proposed solutions (between 2 BPO issues) that would fix it but we got distracted before the matter was settled.
This is not a hard technical problem, although designing a future proof API might be harder.
PyObjC also has Objective-C proxy classes for generic Python objects, making it possible to pass a normal Python dictionary to an Objective-C API that expects an NSDictionary instance.
Also super cool. How similar is this to Jython and IronPython?
I don’t know, I guess this is similar to how those projects proxy between their respective host languages and Python.
Things get interesting when combining the two with sub-interpreters: With the current implementation the Objective-C world would be a channel for passing “live” Python objects between sub-interpreters.
+1
The translation tables for looking up existing proxies (mapping from Python to Objective-C and vice versa) are currently singletons.
This is probably fixable with another level of administration, by keeping track of the sub-interpreter that owns a Python object I could ensure that Python objects owned by a different sub-interpreter are proxied like any other Objective-C object which would close this loophole. That would require significant changes to a code base that’s already fairly complex, but should be fairly straightforward.
Do you think there are any additions we could make to the C-API (more than have been done recently, e.g. PEP 573) that would make this easier. From what I understand, this pattern of a cache/table of global Python objects is a relatively common one. So anything we can do to help transition these to per-interpreter would be broadly beneficial. Ideally it would be done in the least intrusive way possible, reducing churn and touch points. (e.g. a macro to convert existing tables,etc. + an init func to call during module init.)
I can probably fix this entirely on my end: - Use PEP 573 to move the translation tables to per-interpreter storage - Store the sub-interpreter in the ObjC proxy object for Python objects, both to call back into the right subinterpreter in upcalls and to tweak the way “foreign” objects are proxied into a different subinterpreter. But once again, that’s without trying to actually do the work. As usual the devil’s in the details.
Also, FWIW, I've been thinking about possible approaches where the first/main interpreter uses the existing static types, etc. and further subinterpreters use a heap type (etc.) derived mostly automatically from the static one. It's been on my mind because this is one of the last major hurdles to clear in the CPython code before we can make the GIL per-interpreter.
What additional API would be needed?
See above, the main problem is PyGILState_Ensure. I haven’t spent a lot of time thinking about this though, I might find other issues when I try to support sub-interpreters.
Any feedback on this would be useful.
As far as I understand proper support for subinterpreters also requires moving away from static type definitions to avoid sharing objects between interpreters (that is, use the PyType_FromSpec to build types).
Correct, though that is not technically a problem until we stop sharing the GIL.
Right. But a major selling point of sub-interpreters is that this provide a way forward towards having multiple Python threads that don’t share a GIL.
IMHO it would be better to first work out what’s needed to get there, and in particular what changes are needed in extensions. Otherwise extensions may have to be changed multiple times.
Yeah, I see what you're saying. It's been a hard balance to strike. There are really 2 things that have to be done: move all global state to per-interpreter and deal with static types (etc.). Do you think both will require significant work in the community? My instinct, partly informed by my work in CPython along these lines, is that the former is more work and sometimes trickier. The latter is fairly straightforward and much more of an opportunity for automatic approaches.
I don’t particularly like PyType_FromSpec, but transitioning to it should be straightforward except for limitations to that API. Those limitations could be fixed of course. I don’t know how much the move of global state to per-interpreter state affects extensions, other than references to singletons and static types. But with some macro trickery that could be made source compatible for extensions.
At first glance this API does not support everything I do in PyObjC (fun with metaclasses, in C code).
What specific additions/changes would you need?
At least:
- A variant of PyGILState_Ensure that supports sub-interpreters - Defining subclasses of built-in types using PyType_FromSpec, in particular a subclass of “type”.
Thanks!
BTW. In my first mail I mentioned I don’t have a use cases for subinterpreters. I might have a limited use case in the PyObjC domain: implementing plugin bundles for Objective-C applications in Python. These currently share the same interpreter, which can cause problems. Subinterpreters could be helpful there to isolate code, but that would require having an API that conditionally initialises the Python runtime (similar to PyGILState_Ensure, but for the runtime itself).
Ah, interesting.
This wouldn’t fix all problems because you can’t have two different python versions in one proces, but would be better than the status quo.
FWIW, I've been involved in past discussions about the idea of supporting multiple Python runtimes (including difference versions) in the same process. :) On top of that, I know someone who demonstrated to me how to use dlmopen() to do something like this right now. :)
Hmm….. macOS doesn’t support dlmopen, but does have a mechanism to load different versions of libraries (two-level namespaces). It might be interesting to try to use that to load multiple python versions in the same proces (or even the same version multiple times). Ronald