[Import-SIG] On singleton modules, heap types, and subinterpreters

[Petr Viktorin]

On Mon, Jul 27, 2015 at 4:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 July 2015 at 16:44, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Petr Viktorin schrieb am 26.07.2015 um 12:39:
>>> This is a follow-up to PEP 489 and discussions regarding per-module
>>> data PyState_FindModule.
>>> It turned out to be quite the rabbit hole. Apologies for the long
>>> mail, I hope it ends up sufficiently clear.
>>>
>>> Using single-phase initialization (the pre-PEP 489 solution),
>>> extension modules are effectively singletons – there's up to one
>>> instance of a particular module in any given subinterpreter. Cython
>>> modules only allow one instance *per process*.
>>>
>>> Using the new multiple-phase init, one can create several modules from
>>> one PyModuleDef – either (again) one per subinterpreter, or for
>>> testing purposes. As per the goal of PEP 489, this brings extension
>>> modules closer to how Python modules behave.
>>>
>>> The problem is that classes defined in a module don't have a reference
>>> to the module object.
>>> [lots of of other tricky details stripped]
>>
>> Sorry for cutting it short here, but isn't this a hint that linking the new
>> initialisation to subinterpreter support might not be a good idea? I mean,
>> there are a couple of advantages of the new initialisation scheme, e.g. for
>> relative imports etc., which are completely unrelated to subinterpreters.
>> And yet, the PEP suggests that supporting the new module setup scheme
>> should indicate that subinterpreters are supported as well. Given how
>> complex the support seems to be for any non-trivial module, linking the two
>> use cases might end up preventing us from getting any benefit at all out of
>> this for quite some time.
>
> It shouldn't be too complicated when there aren't any custom slot
> implementations involved - regular methods would just get a slightly
> different signature where the defining class is also passed in.
>
> However, I still suspect you're right that we'll end up wanting to
> offer "singleton mode" to allow folks to do a two step upgrade of
> extension modules, first to multi-phase initialisation, and then to
> supporting subinterpreters and reloading.

Right, I'm starting to regret dropping the singleton flag so easily.
But, for better or for worse, both singleton modules and a
PyState_FindModule replacement is now material for 3.6.