[Import-SIG] PEP 489: Redesigning extension module loading

Petr Viktorin encukou at gmail.com
Fri Apr 17 10:40:00 CEST 2015

On 04/17/2015 08:49 AM, Stefan Behnel wrote:
> Petr Viktorin schrieb am 16.04.2015 um 13:05:
>> Extension modules that support the new initialization scheme must export
>> the public symbol "PyModuleExport_<modulename>", where "modulename"
>> is the name of the module. (For modules with non-ASCII names the symbol name
>> is slightly different, see "Export Hook Name" below.)
>> If defined, this symbol must resolve to a C function with the following
>> signature::
>>      PyModuleExport* (*PyModuleExportFunction)(void)
>> The function must return a pointer to a PyModuleExport structure.
>> This structure must be available for the lifetime of the module created from
>> it – usually, it will be declared statically.
>> The PyModuleExport structure describes the new module, similarly to
>> PEP 384's PyType_Spec for types. The structure is defined as::
>>      typedef struct {
>>          int slot;
>>          void *value;
>>      } PyModuleDesc_Slot;
>>      typedef struct {
>>          const char* doc;
>>          int flags;
>>          PyModuleDesc_Slot *slots;
>>      } PyModuleDesc;
> PyModuleDesc or -Export?

Sorry, missed this. Export is a better name, I'll do a replace.

>> Py_mod_statedef
>> ...............
>> The Py_mod_statedef slot is used to allocate per-module storage for C-level
>> state.
> Add a note here that the import machinery will create a normal
> types.ModuleType instance if this setup is used. However:
>> .. note::
>>      If PyModuleDef is reused, this information is taken from PyModuleDef,
>>      so the slot is not necessary.
> Yes, I would really prefer reuse here.

I'll fix the PEP's minor issues, and take this point to python-dev for 

>> Execution slots
>> ---------------
>> These slots may be specified multiple times, and are processed in the order
>> they appear in the slots array.
> Interesting. Not sure if this is useful for anything, but why not. I don't
> think it hurts, and it fits the idea of a slot list.

This is an easy to understand answer to the question "in what order do 
things happen?", which is bound to come up if more advanced slots are 
added. Perhaps I should write out a rule of thumb this implies: "always 
put Py_mod_exec last".
As for allowing repeated slots: it certainly doesn't hurt, and I think 
checking for this case is unnecessary, as is making it undefined behavior.

>> When using the default importmachinery, these slots are processed after
>> import-related attributes specified in PEP 451 [#pep-0451-attributes]_
>> (such as ``__name__`` or ``__loader__``) are set and the module is added
>> to sys.modules.
> What would be a non-default import machinery?

Directly calling Loader.create_module/Loader.execute_module, or other 
such lower-level tools.

>> Py_mod_exec
>> ...........
>> The function in this slot must have the signature::
> Better: "The value (or entry?) in this slot must point to a function with
> the following signature".


>> Legacy Init
>> -----------
>> If the PyModuleExport function is not defined, the import machinery will try to
>> initialize the module using the PyInit hook, as described in PEP 3121.
>> If PyModuleExport is defined, PyModuleInit will be ignored.
> "PyModuleExportFunction", and also use "PyInit" in the last sentence to
> avoid ambiguity. "PyModuleInit" isn't a name that's being used anywhere
> else, I think. (It occurs a couple of times in this PEP.)


>> Multiple modules in one library
>> -------------------------------
>> To support multiple Python modules in one shared library, the library can
>> export additional PyModuleExport* symbols besides the one that corresponds
>> to the library's filename.
>> Note that this mechanism can currently only be used to *load* extra modules,
>> not to *find* them.
>> Given the filesystem location of a shared library and a module name,
>> a module may be loaded with::
>>      import importlib.machinery
>>      import importlib.util
>>      loader = importlib.machinery.ExtensionFileLoader(name, path)
>>      spec = importlib.util.spec_from_loader(name, loader)
>>      return importlib.util.module_from_spec(spec)
>> On platforms that support symbolic links, these may be used to install one
>> library under multiple names, exposing all exported modules to normal
>> import machinery.
> This makes me think, if a module wants to import another one from the same
> shared library (usually in Py_mod_exec), how would that work? Would it
> still require a symlink and full module discovery? Or would a call to
> PyModule_Create(PyModuleDef) be enough? The latter would then have to run
> the complete module initialisation that this PEP defines. That sounds like
> the right way to do it, but it should be mentioned somewhere in this PEP, I
> think.

No, PyModule_Create can't call the whole import machinery. It doesn't 
have the ModuleSpec to pass to the create slot.
Maybe I should make that explicit, and having PyModule_Create fail when 
the def contains slots.

As for the use case, I'm inclined to leave it out of this PEP. I'm 
interested in exploring it later, maybe adding slots for submodules to 
allow C libraries.
But now it seems that this use case is mainly interesting for Cython, 
and there you can do the equivalent of the Python code above.

>> Testing and initial implementations
>> -----------------------------------
>> The ``_csv`` and ``readline`` modules will be converted to the new API as
>> part of the initial implementation.
> Ah, yes, good idea. While not required, I think it's good to have some real
> example in the stdlib, both for exercising the new implementation and for
> others to look at.
>> Possible Future Extensions
>> ==========================
>> [...]
> Good examples. And, if reloading can ever be made to work, maybe just on
> some platforms, this would be the place to provide the entry point.
> Nice work overall, thanks!
> Stefan

More information about the Import-SIG mailing list