On Sat, 4 May 2019 at 06:54, Neil Schemenauer <nas-python@arctrix.com> wrote:
Thanks for the clarity. So the options seem to be:
A) don't support multiple interpreters per process
B) use TLS or some similar mechanism
C) explicitly pass a context pointer to every single CPython API
It's likely we all agree that A is not what we want. We are already trying to support multiple interpreters, even though the current implementation has a number of issues. I guess we could decide "forget it" and rip that all out. Eric Snow would be sad. ;-P
Regarding B vs C, is the only reason to prefer C due to performance? If so, I think it is not worth the pain of it. Based on my small amount of knowledge, TLS can be really cheap. I found this article that has some benchmarks:
https://david-grs.github.io/tls_performance_overhead_cost_linux/Again, TLS is a platform specific mechanism so maybe it is worse on Windows, for example. However, it seems to me that there should be some way to do it with relatively small overhead.
Passing context everywhere doesn't come for free either. You are probably using up another register (or pushing something on the stack, depending on the ABI). If you don't need the context often and you don't switch threads often, option B is likely faster.
I've discussed this with Eric Snow a bit here at PyCon, and I take the view that trying to migrate away from relying on TLS at this late date isn't going to be worth the hassle.
Firstly, extension modules need an implicit context no matter what, and if an implicit context is available, it's far more user friendly if CPython just uses it rather than requiring the extension module to retrieve it from wherever it is stored and then pass it in.
Within CPython, we'd also need to change every API that runs Python code, and every API that may implicitly call back in to Python code, to accept a context parameter.
And for embedding applications, having two ways to do it (ThreadState_Swap vs whatever the new way looks like) will mostly be annoying churn between two functionally equivalent ways of doing things rather than helpful enablement of any actually new functionality.
There may be some new APIs where it makes sense for us to define them as working on a supplied thread state rather than being dependent on the active one, but overall we shouldn't we attempting to change the way we handle the active context.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia