[Import-SIG] PEP 489: Redesigning extension module loading

Stefan Behnel stefan_ml at behnel.de
Fri Apr 17 08:49:56 CEST 2015

Petr Viktorin schrieb am 16.04.2015 um 13:05:
> Extension modules that support the new initialization scheme must export
> the public symbol "PyModuleExport_<modulename>", where "modulename"
> is the name of the module. (For modules with non-ASCII names the symbol name
> is slightly different, see "Export Hook Name" below.)
> If defined, this symbol must resolve to a C function with the following
> signature::
>     PyModuleExport* (*PyModuleExportFunction)(void)
> The function must return a pointer to a PyModuleExport structure.
> This structure must be available for the lifetime of the module created from
> it – usually, it will be declared statically.
> The PyModuleExport structure describes the new module, similarly to
> PEP 384's PyType_Spec for types. The structure is defined as::
>     typedef struct {
>         int slot;
>         void *value;
>     } PyModuleDesc_Slot;
>     typedef struct {
>         const char* doc;
>         int flags;
>         PyModuleDesc_Slot *slots;
>     } PyModuleDesc;

PyModuleDesc or -Export?

> Py_mod_statedef
> ...............
> The Py_mod_statedef slot is used to allocate per-module storage for C-level
> state.

Add a note here that the import machinery will create a normal
types.ModuleType instance if this setup is used. However:

> .. note::
>     If PyModuleDef is reused, this information is taken from PyModuleDef,
>     so the slot is not necessary.

Yes, I would really prefer reuse here.

> Execution slots
> ---------------
> These slots may be specified multiple times, and are processed in the order
> they appear in the slots array.

Interesting. Not sure if this is useful for anything, but why not. I don't
think it hurts, and it fits the idea of a slot list.

> When using the default import machinery, these slots are processed after
> import-related attributes specified in PEP 451 [#pep-0451-attributes]_
> (such as ``__name__`` or ``__loader__``) are set and the module is added
> to sys.modules.

What would be a non-default import machinery?

> Py_mod_exec
> ...........
> The function in this slot must have the signature::

Better: "The value (or entry?) in this slot must point to a function with
the following signature".

> Legacy Init
> -----------
> If the PyModuleExport function is not defined, the import machinery will try to
> initialize the module using the PyInit hook, as described in PEP 3121.
> If PyModuleExport is defined, PyModuleInit will be ignored.

"PyModuleExportFunction", and also use "PyInit" in the last sentence to
avoid ambiguity. "PyModuleInit" isn't a name that's being used anywhere
else, I think. (It occurs a couple of times in this PEP.)

> Singleton Modules
> -----------------

I'll respond to this in a separate email as this seems to need some more

> Multiple modules in one library
> -------------------------------
> To support multiple Python modules in one shared library, the library can
> export additional PyModuleExport* symbols besides the one that corresponds
> to the library's filename.
> Note that this mechanism can currently only be used to *load* extra modules,
> not to *find* them.
> Given the filesystem location of a shared library and a module name,
> a module may be loaded with::
>     import importlib.machinery
>     import importlib.util
>     loader = importlib.machinery.ExtensionFileLoader(name, path)
>     spec = importlib.util.spec_from_loader(name, loader)
>     return importlib.util.module_from_spec(spec)
> On platforms that support symbolic links, these may be used to install one
> library under multiple names, exposing all exported modules to normal
> import machinery.

This makes me think, if a module wants to import another one from the same
shared library (usually in Py_mod_exec), how would that work? Would it
still require a symlink and full module discovery? Or would a call to
PyModule_Create(PyModuleDef) be enough? The latter would then have to run
the complete module initialisation that this PEP defines. That sounds like
the right way to do it, but it should be mentioned somewhere in this PEP, I

> Testing and initial implementations
> -----------------------------------
> The ``_csv`` and ``readline`` modules will be converted to the new API as
> part of the initial implementation.

Ah, yes, good idea. While not required, I think it's good to have some real
example in the stdlib, both for exercising the new implementation and for
others to look at.

> Possible Future Extensions
> ==========================
> [...]

Good examples. And, if reloading can ever be made to work, maybe just on
some platforms, this would be the place to provide the entry point.

Nice work overall, thanks!


More information about the Import-SIG mailing list