On 2018-09-12, Victor Stinner wrote:
About correctness, yesterday, we discussed adding a new first "ctx" parameter to all functions of the C API to support Eric Snow's Sub-Interpreters. [..] At the point, I'm not fully excited by this change, since every C function using the C API would have to pass ctx everywhere.
As I said to you early in the week, you are a brave person to try to tackle this C-API project. Handling the multi-interpreter problem makes harder yet and more invasive. Personally, I think the correct way to do it is to pass ctx as the first parameter to all C-API functions. If it is not a global, I think you have to pass it. Other schemes are suboptimal and we should do the correct thing.
You can have two sets of APIs. One set has ctx as the first argument of every API call. You make a second "easy" set that builds on the first set and uses a global as the ctx argument. This second set could be compatible with the current API. The painful part is that all CPython internals must use the first set of APIs (explicitly pass ctx). Something like Coccinelle could do the hard work.
After you do it, merging between cpython versions becomes essentially impossible since you are changing like 50% of the lines of source code. So, I can imagine that some core developers would resist the change. I think even though it is painful, it is the correct thing to do. If you want to allow Python to be embedded properly (e.g. for game scripting), you have to do it. So, I think we should plan to "bit the bullet" and have a "flag day". I think Python is losing to Lua in these applications because embedding Python doesn't work properly.
Regards,
Neil