Thanks for your input. I now see how things evolved to the present state. in the context of PEP 451, my proposal would have been to move all default module creation tasks to ModuleType.tp_new (taking an optional spec parameter), making separate create and exec unnecessary. Too late, I guess.

Once the question is narrowed down to "How can an extension module fully support subinterpreters and multiple Py_Initialize/Finalize cycles without incurring PEP 3121's performance overhead?" then the short answer becomes "We don't know, but ideas for that are certainly welcome, either here or over on import-sig".

I mentioned the way to avoid state access overhead in my first post. It's independent of module loading mechanism: 1) define a new "calling convention" flag like METH_GLOBALS. 2) store module ref in PyCFunctionObject.m_module (currently it stores only the module name) 3) pass module ref as an extra arg to methods with METH_GLOBALS flag. 4) PyModule_State, reimplemented as a macro, would amount to one indirection from the passed parameter. I suspect that most C ABIs allow to pass the extra arg unconditionally, (this is certainly the case for x86 and x64 on Windows and Linux). Meaning that METH_GLOBALS won't increase the actual number of possible dispatch targets in PyCFunction_Call and won't impact Python-to-C call performance at all.

On 04/15/2016 10:59 AM, Nikita Nemkin wrote:

Thanks for your input. I now see how things evolved to the present state.

in the context of PEP 451, my proposal would have been to move all default module creation tasks to ModuleType.tp_new (taking an optional spec parameter), making separate create and exec unnecessary. Too late, I guess.

...

Once the question is narrowed down to "How can an extension module fully support subinterpreters and multiple Py_Initialize/Finalize cycles without incurring PEP 3121's performance overhead?" then the short answer becomes "We don't know, but ideas for that are certainly welcome, either here or over on import-sig".

I mentioned the way to avoid state access overhead in my first post. It's independent of module loading mechanism:

1) define a new "calling convention" flag like METH_GLOBALS. 2) store module ref in PyCFunctionObject.m_module (currently it stores only the module name)

Wouldn't that break backwards compatibility, though?

3) pass module ref as an extra arg to methods with METH_GLOBALS flag. 4) PyModule_State, reimplemented as a macro, would amount to one indirection from the passed parameter.

I suspect that most C ABIs allow to pass the extra arg unconditionally, (this is certainly the case for x86 and x64 on Windows and Linux). Meaning that METH_GLOBALS won't increase the actual number of possible dispatch targets in PyCFunction_Call and won't impact Python-to-C call performance at all.

My planned approach is a bit more flexible: - Add a reference to the module (ht_module) to heap types - Create a calling convention METH_METHOD, where methods are passed the class that defines the method (which might PyTYPE(self) or a superclass of it) This way methods can get both module state and the class they are defined on, and the replacement for PyModule_State is two indirections. Still, both approaches won't work with slot methods (e.g. nb_add), where there's no space in the API to add an extra argument. Nick proposed a solution in import-sig [0], which is workable but not elegant. But, I think: - METH_METHOD would be useful even if it doesn't solve the problem with slot methods. - A good solution to the slot methods problem is unlikely to render METH_METHOD obsolete. so perhaps solving the 90% case first would be OK. [0] https://mail.python.org/pipermail/import-sig/2015-July/001035.html

Let's move the discussion to import-sig, as Nick explained in the other subthread. Please drop python-ideas from CC when you reply. On 04/15/2016 12:58 PM, Nikita Nemkin wrote:

On Fri, Apr 15, 2016 at 2:24 PM, Petr Viktorin wrote:

...

...

1) define a new "calling convention" flag like METH_GLOBALS. 2) store module ref in PyCFunctionObject.m_module (currently it stores only the module name)

Wouldn't that break backwards compatibility, though?

It will, and I consider this level of breakage acceptable. Alternatively, another field can be added to this struct.

...

My planned approach is a bit more flexible: - Add a reference to the module (ht_module) to heap types - Create a calling convention METH_METHOD, where methods are passed the class that defines the method (which might PyTYPE(self) or a superclass of it)

This way methods can get both module state and the class they are defined on, and the replacement for PyModule_State is two indirections.

I've read the linked import-sig thread and realized the depth of issues involved... Those poor heap type methods don't even have access their own type pointer! In the light of that, your variant is more useful than mine.

Still, without a good slot support option, new METH_X conventions don't look attractive at all. Such fundamental change, but only solves half of the problem.

Well, it solves the problem for methods that have calling conventions, and I'm pretty sure by now that a full solution will need this *plus* another solution for slots. So I'm looking at the problems as two separate parts, and I also think that when it comes to writing PEPs, having two separate PEPs would make this more understandable.

Also, MRO walking is actually not as slow as it seems. Checking Py_TYPE(self) and Py_TYPE(self)->tp_base *inline* will minimize performance overhead in the (very) common case.

I think so as well. This would mean that module state access in named methods is fast; in slot methods it's possible (and usually fast *enough*), and the full solution with __typeslots__ would still be possible.

If non-slot methods had a suitable static anchor (the equivalent of slot function address for slots), they could use MRO walking too.

I think a new METH_* calling style and explicit pointers is a better alternative here.