[Import-SIG] PEP 489: Multi-phase extension module initialization; version 5

[Petr Viktorin]

On 05/20/2015 02:33 AM, Eric Snow wrote:
> On Tue, May 19, 2015 at 8:40 AM, Petr Viktorin <pviktori at redhat.com> wrote:
>> Here is an overview of how the modified importers will operate.
>> Details such as logging or handling of errors and invalid states
>> are left out, and C code is presented with a concise Python-like syntax.
>>
>> The framework that calls the importers is explained in PEP 451
>> [#pep-0451-loading]_.
> 
> I know.  I wrote that PEP. :)
> 
>>
>> importlib/_bootstrap.py:
>>
>>     class BuiltinImporter:
>>         def create_module(self, spec):
>>             module = _imp.create_builtin(spec)
>>
>>         def exec_module(self, module):
>>             _imp.exec_dynamic(module)
>>
>>         def load_module(self, name):
>>             # use a backwards compatibility shim
>>             _load_module_shim(self, name)
> 
> Won't frozen modules be likewise affected?

No, frozen modules are Python source, just not loaded from a file.

[...]
>> Python/import.c (the _imp module):
>>
>>     def create_dynamic(spec):
>>         name = spec.name
>>         path = spec.origin
>>
>>         # Find an already loaded module that used single-phase init.
>>         # For multi-phase initialization, mod is NULL, so a new module
>>         # is always created.
>>         mod = _PyImport_FindExtensionObject(name, name)
>>         if mod:
>>             return mod
>>
>>         return _PyImport_LoadDynamicModuleWithSpec(spec)
>>
>>     def exec_dynamic(module):
>>         def = PyModule_GetDef(module)
> 
> This is the point where custom module types get ignored, right?

Yes. The  actual code has a check for non-modules, to skip exec_dynamic
rather than have PyModule_GetDef raise. I'll add this to the overview to
make things clearer.

>>         state = PyModule_GetState(module)
>>         if state is NULL:
>>             PyModule_ExecDef(module, def)
> 
> Ah, it is idempotent.

Yes, this is the part that disables reload().

[...]
> It may also be worth outlining how PyModuleDef_Init will work.

That's hard to do in Python syntax, since most of what it does is ensure
the def is a valid PyObject. I'll explain it in a different section.
It's a very small, idempotent function:

PyObject*
PyModuleDef_Init(struct PyModuleDef* def)
{
    if (def->m_base.m_index == 0) {
        max_module_number++;
        Py_REFCNT(def) = 1;
        Py_TYPE(def) = &PyModuleDef_Type;
        def->m_base.m_index = max_module_number;
    }
    return (PyObject*)def;
}

The code is lifted straight from PyModule_Create2.

The m_index is bookkeeping for for PyState_FindModule, so it's unused
for modules with multi-phase init, but I didn't want to break the
invariant that it's set up together with Py_TYPE.

-- 
Petr Viktorin